[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 563149840 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003301] x2apic enabled [ 0.005004] Switched APIC routing to physical x2apic. [ 0.006015] kvm-guest: setup PV IPIs [ 0.009490] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.012129] LSM: Security Framework initializing [ 0.013048] Yama: becoming mindful. [ 0.014037] SELinux: Initializing. [ 0.015058] *** VALIDATE selinux *** [ 0.022750] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026764] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028132] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030080] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031122] *** VALIDATE tmpfs *** [ 0.032559] *** VALIDATE proc *** [ 0.033294] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039025] Spectre V2 : User space: Vulnerable [ 0.040005] Speculative Store Bypass: Vulnerable [ 0.043573] debug: unmapping init [mem 0xffffffff9fe59000-0xffffffff9fe60fff] [ 0.045219] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046669] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047016] ... version: 2 [ 0.048006] ... bit width: 48 [ 0.049008] ... generic registers: 4 [ 0.050017] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053009] ... event mask: 000000070000000f [ 0.054308] rcu: Hierarchical SRCU implementation. [ 0.056959] smp: Bringing up secondary CPUs ... [ 0.057863] x86: Booting SMP configuration: [ 0.058051] .... node #0, CPUs: #1 #2 #3 [ 0.065020] smp: Brought up 1 node, 4 CPUs [ 0.067013] smpboot: Max logical packages: 1 [ 0.068013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125011] node 0 deferred pages initialised in 55ms [ 0.129208] devtmpfs: initialized [ 0.131301] x86/mm: Memory block size: 128MB [ 0.135141] gcov: version magic: 0x41383552 [ 0.137302] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144255] pinctrl core: initialized pinctrl subsystem [ 0.146356] [ 0.146966] ************************************************************* [ 0.150028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153011] ** ** [ 0.156012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159016] ** ** [ 0.162010] ** This means that this kernel is built to expose internal ** [ 0.164012] ** IOMMU data structures, which may compromise security on ** [ 0.168013] ** your system. ** [ 0.171012] ** ** [ 0.174012] ** If you see this message and you are not debugging the ** [ 0.176012] ** kernel, report this immediately to your vendor! ** [ 0.179013] ** ** [ 0.181008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184011] ************************************************************* [ 0.187715] NET: Registered protocol family 16 [ 0.189340] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195127] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198451] cpuidle: using governor menu [ 0.199810] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200361] PCI: Using configuration type 1 for base access [ 0.201125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209105] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218078] cryptd: max_cpu_qlen set to 1000 [ 0.220229] ACPI: Added _OSI(Module Device) [ 0.221000] ACPI: Added _OSI(Processor Device) [ 0.223014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227946] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232362] ACPI: Interpreter enabled [ 0.233035] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233964] ACPI: Using IOAPIC for interrupt routing [ 0.235088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239865] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.251127] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.254046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268000] acpiphp: Slot [2] registered [ 0.269117] acpiphp: Slot [5] registered [ 0.271112] acpiphp: Slot [6] registered [ 0.272106] acpiphp: Slot [7] registered [ 0.274086] acpiphp: Slot [8] registered [ 0.275167] acpiphp: Slot [9] registered [ 0.277131] acpiphp: Slot [10] registered [ 0.280103] acpiphp: Slot [3] registered [ 0.281089] acpiphp: Slot [4] registered [ 0.283111] acpiphp: Slot [11] registered [ 0.284095] acpiphp: Slot [12] registered [ 0.286146] acpiphp: Slot [13] registered [ 0.288126] acpiphp: Slot [14] registered [ 0.290088] acpiphp: Slot [15] registered [ 0.291081] acpiphp: Slot [16] registered [ 0.293097] acpiphp: Slot [17] registered [ 0.295096] acpiphp: Slot [18] registered [ 0.298144] acpiphp: Slot [19] registered [ 0.301126] acpiphp: Slot [20] registered [ 0.303109] acpiphp: Slot [21] registered [ 0.304086] acpiphp: Slot [22] registered [ 0.305117] acpiphp: Slot [23] registered [ 0.307075] acpiphp: Slot [24] registered [ 0.308094] acpiphp: Slot [25] registered [ 0.309075] acpiphp: Slot [26] registered [ 0.310000] acpiphp: Slot [27] registered [ 0.310000] acpiphp: Slot [28] registered [ 0.312074] acpiphp: Slot [29] registered [ 0.313064] acpiphp: Slot [30] registered [ 0.314069] acpiphp: Slot [31] registered [ 0.315070] PCI host bridge to bus 0000:00 [ 0.316065] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.322016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.328165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.330978] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.334355] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344956] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.351061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.354018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.363738] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.366032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.368751] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.388015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.393013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.398339] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.408014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.422016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.431000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.431000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.431000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.440012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.456368] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.474016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.487015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.517424] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.524000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.527014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.566015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.584781] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.598021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.613018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.665014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.685097] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.693016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.700015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.740050] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.767827] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.771774] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.774432] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.776327] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.778165] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.784024] iommu: Default domain type: Passthrough [ 0.786000] SCSI subsystem initialized [ 0.788194] ACPI: bus type USB registered [ 0.790087] usbcore: registered new interface driver usbfs [ 0.792123] usbcore: registered new interface driver hub [ 0.794141] usbcore: registered new device driver usb [ 0.796172] pps_core: LinuxPPS API ver. 1 registered [ 0.798011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.801056] PTP clock support registered [ 0.805088] EDAC MC: Ver: 3.0.0 [ 0.806772] PCI: Using ACPI for IRQ routing [ 0.808984] NetLabel: Initializing [ 0.810009] NetLabel: domain hash size = 128 [ 0.812011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.814078] NetLabel: unlabeled traffic allowed by default [ 0.817197] vgaarb: loaded [ 0.818667] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.820010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.828197] clocksource: Switched to clocksource kvm-clock [ 0.952330] VFS: Disk quotas dquot_6.6.0 [ 0.953987] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.956309] *** VALIDATE ramfs *** [ 0.958755] *** VALIDATE hugetlbfs *** [ 0.960524] pnp: PnP ACPI init [ 0.963949] pnp: PnP ACPI: found 6 devices [ 0.982336] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.985949] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.987705] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.990031] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.992845] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.995633] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.998753] NET: Registered protocol family 2 [ 1.001346] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.006978] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.011462] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.017169] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.020625] TCP: Hash tables configured (established 65536 bind 65536) [ 1.023337] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.026924] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.031582] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.035200] NET: Registered protocol family 1 [ 1.037865] RPC: Registered named UNIX socket transport module. [ 1.039928] RPC: Registered udp transport module. [ 1.043765] RPC: Registered tcp transport module. [ 1.045773] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.048486] NET: Registered protocol family 44 [ 1.050615] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.052796] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.055186] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.058061] PCI: CLS 0 bytes, default 64 [ 1.060083] Unpacking initramfs... [ 2.986550] debug: unmapping init [mem 0xffff92487cc54000-0xffff92487ffbffff] [ 2.994241] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.997173] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.002506] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.599845] Initialise system trusted keyrings [ 3.601896] Key type blacklist registered [ 3.604064] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.614535] zbud: loaded [ 3.617955] *** VALIDATE nfs *** [ 3.619601] *** VALIDATE nfs4 *** [ 3.621571] pstore: using deflate compression [ 3.626118] Platform Keyring initialized [ 3.759611] NET: Registered protocol family 38 [ 3.762182] Key type asymmetric registered [ 3.764362] Asymmetric key parser 'x509' registered [ 3.766642] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.770717] io scheduler mq-deadline registered [ 3.772858] io scheduler kyber registered [ 3.775249] io scheduler bfq registered [ 3.777648] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.781744] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.785106] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.788772] ACPI: Power Button [PWRF] [ 3.795324] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.803226] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.834213] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.848220] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.042548] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.079610] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.114455] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.121901] Non-volatile memory driver v1.3 [ 4.125213] Linux agpgart interface v0.103 [ 4.162600] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.167239] vda: detected capacity change from 0 to 74686464 [ 4.253110] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.256530] vdb: detected capacity change from 0 to 1073741824 [ 4.278886] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.282857] vdc: detected capacity change from 0 to 2621440000 [ 4.300684] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.304247] vdd: detected capacity change from 0 to 2621440000 [ 4.327248] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.330476] vde: detected capacity change from 0 to 4294967296 [ 4.350911] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.353978] vdf: detected capacity change from 0 to 4294967296 [ 4.360990] libphy: Fixed MDIO Bus: probed [ 4.366825] usbcore: registered new interface driver usbserial_generic [ 4.369055] usbserial: USB Serial support registered for generic [ 4.371771] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.378273] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.380238] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.383293] mousedev: PS/2 mouse device common for all mice [ 4.387515] rtc_cmos 00:05: RTC can wake from S4 [ 4.391987] rtc_cmos 00:05: registered as rtc0 [ 4.394570] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.399822] intel_pstate: CPU model not supported [ 4.404550] hid: raw HID events driver (C) Jiri Kosina [ 4.407693] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.414635] usbcore: registered new interface driver usbhid [ 4.416957] usbhid: USB HID core driver [ 4.419617] drop_monitor: Initializing network drop monitor service [ 4.422689] Initializing XFRM netlink socket [ 4.425748] NET: Registered protocol family 10 [ 4.429152] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.435940] Segment Routing with IPv6 [ 4.438195] NET: Registered protocol family 17 [ 4.441952] mpls_gso: MPLS GSO support [ 4.449589] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.455505] RAS: Correctable Errors collector initialized. [ 4.458212] AVX version of gcm_enc/dec engaged. [ 4.460389] AES CTR mode by8 optimization enabled [ 4.570884] sched_clock: Marking stable (4570864208, 0)->(5737958609, -1167094401) [ 4.574630] registered taskstats version 1 [ 4.576654] Loading compiled-in X.509 certificates [ 4.579190] zswap: loaded using pool lzo/zbud [ 4.619681] Key type big_key registered [ 4.637919] Key type encrypted registered [ 4.640314] ima: No TPM chip found, activating TPM-bypass! [ 4.642890] ima: Allocated hash algorithm: sha1 [ 4.644832] ima: No architecture policies found [ 4.646986] evm: Initialising EVM extended attributes: [ 4.648987] evm: security.selinux [ 4.650571] evm: security.ima [ 4.651799] evm: security.capability [ 4.653309] evm: HMAC attrs: 0x1 [ 4.657589] rtc_cmos 00:05: setting system clock to 2026-09-02 14:22:43 UTC (1788358963) [ 4.674777] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 4.688498] debug: unmapping init [mem 0xffffffff9fb82000-0xffffffff9fe58fff] [ 4.708067] Write protecting the kernel read-only data: 28672k [ 4.719927] debug: unmapping init [mem 0xffffffff9e203000-0xffffffff9e3fffff] [ 4.728609] debug: unmapping init [mem 0xffffffff9eb14000-0xffffffff9ebfffff] [ 4.848831] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.866848] systemd[1]: Detected virtualization kvm. [ 4.869938] systemd[1]: Detected architecture x86-64. [ 4.873225] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.906607] systemd[1]: No hostname configured. [ 4.909088] systemd[1]: Set hostname to . [ 4.912327] random: systemd: uninitialized urandom read (16 bytes read) [ 4.917160] systemd[1]: Initializing machine ID from random generator. [ 5.294297] random: systemd: uninitialized urandom read (16 bytes read) [ 5.297615] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.303244] random: systemd: uninitialized urandom read (16 bytes read) [ 5.306857] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.315572] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.894607] device-mapper: uevent: version 1.0.3 [ 6.896520] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 7.840508] random: fast init done [ 7.855693] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.296983] scsi host0: ata_piix [ 8.327427] scsi host1: ata_piix [ 8.331751] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.343426] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.552434] random: crng init done [ 15.555488] random: 7 urandom warning(s) missed due to ratelimiting [ 18.514909] dracut-initqueue[579]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.790117] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.734157] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.522079] SELinux: Disabled at runtime. [ 23.879460] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.902497] systemd[1]: Detected virtualization kvm. [ 23.906755] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.281794] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.289930] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.307404] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.318557] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.336516] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.364941] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.379520] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ 27.851624] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 28.866556] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.134554] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.292474] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.663110] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.825048] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.780896] Key type dns_resolver registered [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 37.023168] NFS: Registering the id_resolver key type [ 37.026840] Key type id_resolver registered [ 37.032950] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg358-server login: [ 72.742276] hrtimer: interrupt took 6089706 ns [ 128.085235] libcfs: loading out-of-tree module taints kernel. [ 128.179660] Key type ._llcrypt registered [ 128.180760] Key type .llcrypt registered [ 128.337452] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 146.508578] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 147.561069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 147.576749] alg: No test for adler32 (adler32-zlib) [ 148.902322] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 149.509721] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 151.199500] Key type lgssc registered [ 152.842507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 169.337532] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 208.612622] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 221.643676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 221.682927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.942156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 222.996971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 223.117239] Lustre: lustre-MDT0000: new disk, initializing [ 223.230117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.249805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 228.378330] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.910320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.001963] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 241.030964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 241.035140] Lustre: Skipped 1 previous similar message [ 241.099514] Lustre: lustre-MDT0001: new disk, initializing [ 241.151558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.180440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 241.194516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 245.112940] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.575364] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 257.869432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.158190] Lustre: lustre-OST0000: new disk, initializing [ 258.165802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 258.177523] Lustre: 8441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.298728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.722657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 263.732868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 263.833115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 264.331863] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 278.852277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 278.997961] Lustre: lustre-OST0001: new disk, initializing [ 279.001259] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 279.006792] Lustre: 9514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 279.092906] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 285.905329] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.360517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 289.373293] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 289.445712] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 298.312129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 304.626658] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 311.267850] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing check_logdir /tmp/testlogs/ [ 316.222556] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing yml_node [ 320.480070] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 322.891686] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 325.142023] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 326.918454] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 2 10:28:04 EDT 2026 [ 342.914584] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 344.499276] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 361.445473] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.448235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.463488] Lustre: Skipped 3 previous similar messages [ 361.475469] Lustre: Skipped 3 previous similar messages [ 366.319155] Lustre: server umount lustre-MDT0000 complete [ 370.092783] LustreError: 12153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359328 with bad export cookie 16975237516105064606 [ 370.099047] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 370.118518] LustreError: 12153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 371.680179] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.681409] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.697694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.715706] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 371.729294] Lustre: Skipped 1 previous similar message [ 375.712648] LustreError: 12456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.714584] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 375.745370] LustreError: 12456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 375.772938] Lustre: Skipped 1 previous similar message [ 376.734385] Lustre: server umount lustre-MDT0001 complete [ 380.929063] Lustre: server umount lustre-OST0000 complete [ 385.342773] Lustre: server umount lustre-OST0001 complete [ 391.801536] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 400.871341] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 431.156547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.298701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 431.324895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 431.395409] Lustre: lustre-MDT0000: new disk, initializing [ 431.468764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.485839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 434.900931] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 443.685600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.824702] Lustre: 15064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 443.926563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 443.936691] Lustre: Skipped 1 previous similar message [ 444.054507] Lustre: lustre-MDT0001: new disk, initializing [ 444.162824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 444.250404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 444.260975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 447.757451] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 454.855760] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 458.641413] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 463.563373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 463.793976] Lustre: lustre-OST0000: new disk, initializing [ 463.797784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 463.803475] Lustre: 17031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 463.913692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.022415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.033283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 465.121884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 468.827785] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 476.835764] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 480.648478] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 480.906268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 484.491747] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 484.668629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 490.983645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.983658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 490.984809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.995947] Lustre: Skipped 2 previous similar messages [ 491.010692] LustreError: Skipped 1 previous similar message [ 492.478992] Lustre: server umount lustre-OST0000 complete [ 496.097244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.099321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 496.106023] Lustre: Skipped 1 previous similar message [ 496.114942] Lustre: Skipped 2 previous similar messages [ 501.890884] Lustre: server umount lustre-MDT0000 complete [ 504.520421] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359463 with bad export cookie 16975237516105066139 [ 504.525151] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.529819] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 504.712997] Lustre: server umount lustre-MDT0001 complete [ 507.756475] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:31:05 (1788359465) [ 509.724349] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 511.132266] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:31:08 (1788359468) [ 515.934458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.302219] LustreError: 19441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.366706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.349542] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 521.697853] LustreError: 19442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.935267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.966490] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 532.322058] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.000949] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 540.712042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 544.651749] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 549.339193] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 560.611137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 560.616311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.619623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 560.634131] Lustre: Skipped 1 previous similar message [ 560.636728] Lustre: Skipped 2 previous similar messages [ 564.094560] Lustre: server umount lustre-OST0000 complete [ 570.852559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.861491] Lustre: Skipped 1 previous similar message [ 572.510296] Lustre: server umount lustre-MDT0000 complete [ 575.103106] LustreError: 19423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359533 with bad export cookie 16975237516105067021 [ 575.105906] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.118936] LustreError: 19423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 575.396823] Lustre: server umount lustre-MDT0001 complete [ 582.227565] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 584.066452] Key type lgssc unregistered [ 584.278590] LNet: 22970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.284488] LNetError: 22970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.303996] LNet: Removed LNI 192.168.203.158@tcp [ 584.966157] Key type .llcrypt unregistered [ 584.968169] Key type ._llcrypt unregistered [ 599.284312] Key type ._llcrypt registered [ 599.286378] Key type .llcrypt registered [ 599.364493] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 600.007898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 600.048649] alg: No test for adler32 (adler32-zlib) [ 601.032730] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 601.214557] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 602.879222] Key type lgssc registered [ 603.665451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 609.871730] loop: module loaded [ 609.939354] loop0: detected capacity change from 0 to 204800000 [ 610.114605] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.121532] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.311295] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.519493] loop0: detected capacity change from 0 to 204800000 [ 614.688505] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.702400] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.718187] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 618.078351] loop0: detected capacity change from 0 to 204800000 [ 621.908184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 621.932930] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.101526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 623.153852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 623.211929] Lustre: lustre-MDT0000: new disk, initializing [ 623.280226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 623.297398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 626.341652] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 633.006447] loop1: detected capacity change from 0 to 204800000 [ 636.632842] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 636.819628] Lustre: lustre-OST07c6: new disk, initializing [ 636.824286] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 636.828751] Lustre: Skipped 1 previous similar message [ 636.833506] Lustre: 26133:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 636.894445] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 639.798564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 639.806465] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 639.837954] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 641.336065] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 645.644716] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 645.649921] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 645.658645] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 645.665519] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 645.675587] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 645.680098] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 665.878259] Lustre: 25120:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 665.882569] Lustre: 25120:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 665.888364] Lustre: 25120:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 665.893946] Lustre: 25120:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 665.902992] Lustre: 25120:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 665.908086] Lustre: 25120:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.014583] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 670.025353] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 670.029853] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 670.034751] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.039436] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 670.044437] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.049146] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 670.053403] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.058143] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 670.062466] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.066450] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.070473] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.195238] Lustre: 25119:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 673.200545] Lustre: 25119:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 673.204668] Lustre: 25119:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 673.208820] Lustre: 25119:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.212742] Lustre: 25119:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 673.217595] Lustre: 25119:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.222088] Lustre: 25119:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 673.225961] Lustre: 25119:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.230227] Lustre: 25119:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 673.235119] Lustre: 25119:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.239425] Lustre: 25119:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 673.243233] Lustre: 25119:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.441278] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 677.450773] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 677.456836] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 677.461356] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.466987] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 677.471596] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.476263] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 677.483544] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.492620] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 677.499530] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.504357] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 677.509337] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.796757] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 687.801283] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 687.804708] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 687.809179] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 687.814352] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 687.817325] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 687.821411] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 687.825441] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 687.828820] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 687.831585] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 687.834996] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 687.839017] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.251296] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 737.251305] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.268450] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 739.497622] Lustre: server umount lustre-OST07c6 complete [ 744.835513] Lustre: server umount lustre-MDT0000 complete [ 753.464072] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 755.097974] Key type lgssc unregistered [ 755.287159] LNet: 28698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.291805] LNetError: 28698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.314191] LNet: Removed LNI 192.168.203.158@tcp [ 755.866929] Key type .llcrypt unregistered [ 755.868893] Key type ._llcrypt unregistered [ 768.272066] Key type ._llcrypt registered [ 768.275897] Key type .llcrypt registered [ 768.348885] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 769.101891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 769.120467] alg: No test for adler32 (adler32-zlib) [ 770.108785] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 770.298243] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 771.943250] Key type lgssc registered [ 772.870418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 778.403396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 778.419899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.697250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.237651] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 787.619596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 787.862437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 790.404907] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 795.008126] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 798.382814] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 802.466272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 802.744466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.782545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 806.599305] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 811.497066] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 824.806030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 824.807164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 824.809666] LustreError: Skipped 1 previous similar message [ 824.826890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 826.249590] Lustre: server umount lustre-OST0000 complete [ 828.899255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 828.908204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.921760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 833.504989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.506054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 834.705699] Lustre: server umount lustre-MDT0000 complete [ 837.125462] LustreError: 29927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359795 with bad export cookie 15652520889648656192 [ 837.128176] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.138281] LustreError: 29927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 846.413357] Lustre: server umount lustre-MDT0001 complete [ 852.757641] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 854.570466] Key type lgssc unregistered [ 854.759860] LNet: 33678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.764403] LNetError: 33678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.781133] LNet: Removed LNI 192.168.203.158@tcp [ 855.338351] Key type .llcrypt unregistered [ 855.340935] Key type ._llcrypt unregistered [ 867.211680] Key type ._llcrypt registered [ 867.215375] Key type .llcrypt registered [ 867.299594] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 867.978071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 868.002271] alg: No test for adler32 (adler32-zlib) [ 868.928853] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 869.086380] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 870.719197] Key type lgssc registered [ 871.496566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 876.466310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 876.480024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.736240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.091378] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 884.941783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.211261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 887.870650] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 892.123359] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 895.575718] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 900.139930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 900.423592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.517034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 904.015369] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 908.328243] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 917.985325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 917.992196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.001500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 920.035894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 920.036767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 920.048557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.156034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.160570] Lustre: Skipped 2 previous similar messages [ 930.271140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 930.272728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.284160] Lustre: Skipped 1 previous similar message [ 930.367751] Lustre: server umount lustre-OST0000 complete [ 935.393152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.403646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.410461] Lustre: Skipped 1 previous similar message [ 938.749539] Lustre: server umount lustre-MDT0000 complete [ 940.514775] LustreError: 34923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 940.533807] LustreError: 34923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 940.997428] LustreError: 34906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359899 with bad export cookie 13370560787671594357 [ 941.001538] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 941.005941] LustreError: 34906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 941.213256] Lustre: server umount lustre-MDT0001 complete [ 947.484442] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 949.401476] Key type lgssc unregistered [ 949.619487] LNet: 38456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.623157] LNetError: 38456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.632192] LNet: Removed LNI 192.168.203.158@tcp [ 950.149185] Key type .llcrypt unregistered [ 950.150904] Key type ._llcrypt unregistered [ 964.990316] Key type ._llcrypt registered [ 964.992758] Key type .llcrypt registered [ 965.091885] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 965.649295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 965.660431] alg: No test for adler32 (adler32-zlib) [ 966.611869] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 966.798215] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 968.423358] Key type lgssc registered [ 969.136798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.773976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.785762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.045820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.252210] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 981.890926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.155597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.337886] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 988.549996] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 991.908518] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 995.871376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 996.098726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.128482] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 999.206190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1003.030809] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1011.684547] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.685443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1011.689032] Lustre: Skipped 2 previous similar messages [ 1011.690819] Lustre: Skipped 1 previous similar message [ 1014.842921] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 1014.846576] Lustre: Skipped 1 previous similar message [ 1016.803363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.807071] Lustre: Skipped 2 previous similar messages [ 1016.916563] Lustre: server umount lustre-MDT0000 complete [ 1018.651539] LustreError: 39832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788359977 with bad export cookie 6787978007953031284 [ 1018.657574] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1018.660541] LustreError: 39832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1019.950615] LustreError: 42341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.959891] LustreError: 42341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1019.971181] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.58@tcp (stopping) [ 1021.920198] LustreError: 42341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.920693] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.970057] LustreError: 39846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.857222] Lustre: server umount lustre-MDT0001 complete [ 1043.690993] Lustre: server umount lustre-OST0000 complete [ 1052.585426] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1054.325490] Key type lgssc unregistered [ 1054.512826] LNet: 43711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.520401] LNetError: 43711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.534994] LNet: Removed LNI 192.168.203.158@tcp [ 1055.037442] Key type .llcrypt unregistered [ 1055.039382] Key type ._llcrypt unregistered [ 1068.881475] Key type ._llcrypt registered [ 1068.884605] Key type .llcrypt registered [ 1068.944622] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1069.399981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.446744] alg: No test for adler32 (adler32-zlib) [ 1070.357292] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1070.488387] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1072.103163] Key type lgssc registered [ 1072.769445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1077.235521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1077.252690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.524089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.794956] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1085.433390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.661302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1087.931566] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1092.095429] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1095.450743] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1099.082100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.290155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.704062] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1103.403627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1106.636758] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1115.617315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1115.619592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.626748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1115.637338] Lustre: Skipped 1 previous similar message [ 1115.646175] Lustre: Skipped 1 previous similar message [ 1117.429480] Lustre: server umount lustre-OST0000 complete [ 1132.000756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1132.008078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.016917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.096704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.099156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.750230] Lustre: server umount lustre-MDT0000 complete [ 1139.599389] LustreError: 45085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360098 with bad export cookie 14488760229375007318 [ 1139.602632] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1139.608232] LustreError: 45085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1145.807713] Lustre: server umount lustre-MDT0001 complete [ 1151.055053] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1152.549310] Key type lgssc unregistered [ 1152.708348] LNet: 48810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.712153] LNetError: 48810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.721698] LNet: Removed LNI 192.168.203.158@tcp [ 1153.167955] Key type .llcrypt unregistered [ 1153.169682] Key type ._llcrypt unregistered [ 1165.289729] Key type ._llcrypt registered [ 1165.291526] Key type .llcrypt registered [ 1165.353411] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1165.778925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.824024] alg: No test for adler32 (adler32-zlib) [ 1166.743333] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1166.873726] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1168.487295] Key type lgssc registered [ 1168.996662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1172.794886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1172.806965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.045067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.008881] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1180.739132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.005783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1183.201798] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1187.129990] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1189.992098] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1193.595198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1193.773574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.936767] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1198.891534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1200.902736] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1209.087703] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1210.783996] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1221.823428] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0210h42m59s [ 1222.538464] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1223.735950] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 1224.751775] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 1226.722099] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.726393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.731983] Lustre: Skipped 2 previous similar messages [ 1226.734769] Lustre: Skipped 1 previous similar message [ 1229.627952] Lustre: server umount lustre-MDT0000 complete [ 1230.895261] LustreError: 52694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.904312] LustreError: 52694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1231.281419] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1233.889674] LustreError: 50203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.895980] LustreError: 50203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1234.512165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.575588] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.697682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.737556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1235.753777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1236.467214] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1237.983524] Lustre: DEBUG MARKER: Wait for df (36611) ... [ 1238.646388] Lustre: DEBUG MARKER: done [ 1241.060461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.071560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.074615] Lustre: Skipped 5 previous similar messages [ 1246.178855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1246.184030] Lustre: Skipped 1 previous similar message [ 1246.363564] Lustre: server umount lustre-OST0000 complete [ 1251.297912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1251.298126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.307358] Lustre: Skipped 2 previous similar messages [ 1254.332994] Lustre: server umount lustre-MDT0000 complete [ 1256.160910] LustreError: 50184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360215 with bad export cookie 5905200949562469424 [ 1256.166944] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.170097] LustreError: 50184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1256.351370] Lustre: server umount lustre-MDT0001 complete [ 1261.461695] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1262.908453] Key type lgssc unregistered [ 1263.063559] LNet: 55117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.068337] LNetError: 55117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.084073] LNet: Removed LNI 192.168.203.158@tcp [ 1263.574505] Key type .llcrypt unregistered [ 1263.576539] Key type ._llcrypt unregistered [ 1281.959436] Key type ._llcrypt registered [ 1281.961546] Key type .llcrypt registered [ 1282.019490] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1282.504647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1282.517615] alg: No test for adler32 (adler32-zlib) [ 1283.406078] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1283.526526] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1285.127185] Key type lgssc registered [ 1285.686320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1289.386512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1289.398027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.518592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1290.531455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1290.607319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.426353] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1296.270152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.337837] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1296.356749] Lustre: 57312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1296.496470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1298.338506] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1301.540319] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1303.912927] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1306.789665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1306.899395] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1306.984961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1309.590685] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1312.740163] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1312.811190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1317.513888] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0210h44m35s [ 1318.195633] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1321.217851] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1322.422798] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1322.440628] LustreError: 57318:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1342.533812] Lustre: lustre-MDT0000: Client b9389251-608d-467b-a0bd-1c74283f3ffc (at 192.168.203.58@tcp) reconnecting [ 1362.487182] LustreError: 57318:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1363.454965] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1364.483567] Lustre: DEBUG MARKER: done [ 1368.033870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1368.037980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.044791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1369.060867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1369.061393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.072253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1372.448901] Lustre: server umount lustre-OST0000 complete [ 1374.177551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.179178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.185399] Lustre: Skipped 1 previous similar message [ 1374.192411] Lustre: Skipped 2 previous similar messages [ 1380.266481] Lustre: server umount lustre-MDT0000 complete [ 1381.847728] LustreError: 59168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360340 with bad export cookie 4927384492457263184 [ 1381.851689] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1381.853377] LustreError: 59168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1381.998892] Lustre: server umount lustre-MDT0001 complete [ 1386.633552] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1387.983260] Key type lgssc unregistered [ 1388.126149] LNet: 61630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.130607] LNetError: 61630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.145801] LNet: Removed LNI 192.168.203.158@tcp [ 1388.547985] Key type .llcrypt unregistered [ 1388.549828] Key type ._llcrypt unregistered [ 1406.930729] Key type ._llcrypt registered [ 1406.932809] Key type .llcrypt registered [ 1406.977505] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1407.391449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1407.411871] alg: No test for adler32 (adler32-zlib) [ 1408.298500] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1408.412494] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1410.007140] Key type lgssc registered [ 1410.462302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1411.921724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1411.935315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.050645] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1413.064711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1413.145264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.620073] Lustre: Failing over lustre-MDT0000 [ 1413.741864] Lustre: server umount lustre-MDT0000 complete [ 1416.426773] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:46:14 (1788360374) [ 1419.224260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.405273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.139602] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1424.801415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.851881] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1424.865690] Lustre: 64403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1424.976058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1426.561722] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1429.363457] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1431.499978] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1434.212979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.308451] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1434.404324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.965746] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1439.728148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1440.069263] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1444.367698] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1449.954661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.955913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.961503] Lustre: Skipped 2 previous similar messages [ 1449.966451] Lustre: Skipped 2 previous similar messages [ 1452.385611] Lustre: server umount lustre-MDT0000 complete [ 1453.862846] LustreError: 64394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360412 with bad export cookie 16361506454260338369 [ 1453.866846] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1453.868038] LustreError: 64394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1454.030534] Lustre: server umount lustre-MDT0001 complete [ 1455.457118] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1457.253931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1460.902691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.199943] LustreError: 63326:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cd2546e0 x1875231760219392/t0(0) o250->MGC192.168.203.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1463.353845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.386224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1464.853822] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1467.898351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.990766] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.452108] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1470.118366] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1471.135089] Lustre: 63327:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788360413/real 1788360413] req@00000000d2af5ca2 x1875231760219008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788360429 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1472.054289] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1473.183124] Lustre: 63327:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788360416/real 1788360416] req@000000000899da79 x1875231760219136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788360432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1474.172783] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1475.551206] Lustre: 63329:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788360418/real 1788360418] req@000000001d095614 x1875231760219264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788360434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1478.624269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1478.627387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.633249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.199135] Lustre: 63329:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788360422/real 1788360422] req@00000000a19df7b6 x1875231760219520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788360438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1480.672585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1480.673594] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.675183] Lustre: Skipped 1 previous similar message [ 1480.681294] Lustre: Skipped 1 previous similar message [ 1482.267454] Lustre: server umount lustre-MDT0000 complete [ 1483.776636] LustreError: 64393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360442 with bad export cookie 16361506454260340637 [ 1483.779734] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1483.783737] LustreError: 64393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1483.918177] Lustre: server umount lustre-MDT0001 complete [ 1486.482274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1490.121049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.031375] LustreError: 66250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.031738] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1494.039239] LustreError: 66250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1494.065922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.068425] Lustre: Skipped 1 previous similar message [ 1494.096215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1495.590584] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1498.942301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.796127] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1501.159403] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1501.163542] Lustre: Skipped 1 previous similar message [ 1502.175162] Lustre: 63329:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788360444/real 1788360444] req@000000000616e654 x1875231760239872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788360460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1503.971191] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1506.252411] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1509.344713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1509.348912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.356133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.359012] Lustre: Skipped 1 previous similar message [ 1510.367210] Lustre: 63327:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788360452/real 1788360452] req@00000000ac82abd9 x1875231760240384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788360468 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1510.380870] Lustre: 63327:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1514.850271] Lustre: server umount lustre-MDT0000 complete [ 1516.514101] LustreError: 68896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.523251] LustreError: 68896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1516.618523] LustreError: 69790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360475 with bad export cookie 16361506454260342016 [ 1516.621181] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1516.626804] LustreError: 69790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1516.754974] Lustre: server umount lustre-MDT0001 complete [ 1519.510544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1521.029472] Lustre: DEBUG MARKER: files compared the same [ 1532.193496] Lustre: server umount lustre-OST0000 complete [ 1538.645630] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1539.877740] Key type lgssc unregistered [ 1540.024410] LNet: 74115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.027598] LNetError: 74115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.036321] LNet: Removed LNI 192.168.203.158@tcp [ 1540.390066] Key type .llcrypt unregistered [ 1540.391905] Key type ._llcrypt unregistered [ 1550.438970] Key type ._llcrypt registered [ 1550.440235] Key type .llcrypt registered [ 1550.485131] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1550.810133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1550.837048] alg: No test for adler32 (adler32-zlib) [ 1551.708385] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1551.808476] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1553.399200] Key type lgssc registered [ 1553.829510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1556.838101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1556.848206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.051111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.701212] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug +malloc all [ 1562.995091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.115215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1564.649658] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug +malloc all [ 1567.528147] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1569.599926] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1572.443327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.590176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.936822] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug +malloc all [ 1577.788432] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1582.819050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1593.311623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1593.312302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.315142] LustreError: Skipped 1 previous similar message [ 1593.320356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1593.323457] Lustre: Skipped 1 previous similar message [ 1596.683875] Lustre: server umount lustre-OST0000 complete [ 1598.431616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.431847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1598.435856] Lustre: Skipped 2 previous similar messages [ 1603.553063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1603.555829] Lustre: Skipped 2 previous similar messages [ 1608.671724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.674876] Lustre: Skipped 1 previous similar message [ 1612.255194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1612.324487] Lustre: server umount lustre-MDT0000 complete [ 1613.791684] LustreError: 76261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.800127] LustreError: 76261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1613.867187] LustreError: 76255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360572 with bad export cookie 5691449439773490052 [ 1613.869534] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1613.873291] LustreError: 76255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1614.001934] Lustre: server umount lustre-MDT0001 complete [ 1618.117137] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1619.313074] Key type lgssc unregistered [ 1619.446373] LNet: 79074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.449774] LNetError: 79074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.461360] LNet: Removed LNI 192.168.203.158@tcp [ 1619.810904] Key type .llcrypt unregistered [ 1619.812865] Key type ._llcrypt unregistered [ 1631.273361] Key type ._llcrypt registered [ 1631.274914] Key type .llcrypt registered [ 1631.348776] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1632.073712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1632.108352] alg: No test for adler32 (adler32-zlib) [ 1633.041634] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1633.216248] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1634.847115] Key type lgssc registered [ 1636.274473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1644.689874] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1644.702812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.905612] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1666.015980] LustreError: 80443:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.158@tcp: failed processing log, type 4: rc = -110 [ 1692.767579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.373280] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1716.632852] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1725.164848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.310162] Lustre: *** cfs_fail_loc=706, val=0*** [ 1725.713396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.818793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1730.282509] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1733.091994] LustreError: 81527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.113681] LustreError: 81527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1738.209253] LustreError: 80469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1738.223126] LustreError: 80469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1738.335789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.687706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1742.510391] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1748.734959] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1752.957288] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1756.640656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1756.644256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.651700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1759.201844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1759.207098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1759.209656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.536628] Lustre: server umount lustre-OST0000 complete [ 1764.324069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1764.329566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.338288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1764.342338] Lustre: Skipped 1 previous similar message [ 1764.346056] Lustre: Skipped 1 previous similar message [ 1769.367271] Lustre: server umount lustre-MDT0000 complete [ 1769.441557] LustreError: 81533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1769.448656] LustreError: 81533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1772.072413] LustreError: 83304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360730 with bad export cookie 12013772528331515537 [ 1772.078422] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1772.080732] LustreError: 83304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1772.384606] Lustre: server umount lustre-MDT0001 complete [ 1780.533562] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1782.948312] Key type lgssc unregistered [ 1783.192301] LNet: 84055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.201377] LNetError: 84055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.233381] LNet: Removed LNI 192.168.203.158@tcp [ 1785.020174] Key type .llcrypt unregistered [ 1785.025934] Key type ._llcrypt unregistered [ 1805.077658] Key type ._llcrypt registered [ 1805.082589] Key type .llcrypt registered [ 1805.182254] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1806.177643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.235467] alg: No test for adler32 (adler32-zlib) [ 1807.380800] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 1807.610802] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1809.303271] Key type lgssc registered [ 1810.305455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1816.622570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1816.643665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.323335] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1827.399068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.794266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1831.166806] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1832.930800] LustreError: 86137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1832.943653] LustreError: 86137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1837.973599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1838.053251] LustreError: 86136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.069431] LustreError: 86136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1838.302408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.068231] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1843.686852] LustreError: 86526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.718936] LustreError: 86526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1848.803314] LustreError: 86141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.816592] LustreError: 86141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1850.060857] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.089304] Lustre: MGS: Client bed4f4e4-4349-4e49-9899-8aa39c2eb4e3 (at 0@lo) reconnecting [ 1850.111609] LustreError: 86151:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1850.392680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.483881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1854.319695] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1870.816079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1870.825582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.831871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1870.839847] Lustre: Skipped 1 previous similar message [ 1870.850912] Lustre: Skipped 1 previous similar message [ 1873.225572] Lustre: server umount lustre-OST0000 complete [ 1881.059142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.059450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1881.081774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.678236] Lustre: server umount lustre-MDT0000 complete [ 1886.181023] LustreError: 88057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.195847] LustreError: 88057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1886.663917] Lustre: server umount lustre-MDT0001 complete [ 1890.163757] Lustre: server umount MGS complete [ 1899.110800] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:54:16 (1788360856) [ 1922.862332] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 1930.861201] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1958.853711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.585752] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1970.928785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.159510] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1971.227239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1971.295652] Lustre: lustre-MDT0001: new disk, initializing [ 1971.383753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1971.425802] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1971.437101] Lustre: 93306:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1971.452108] LustreError: 93306:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1975.260304] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1984.221670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1984.429391] Lustre: lustre-OST0000: new disk, initializing [ 1984.435574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1984.448278] Lustre: 94071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1984.502642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.850917] Lustre: 94155:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000cb98c262 x1875232178609024/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788360961 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1990.140848] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1998.506259] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2004.268191] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.279902] Lustre: MGS: Client 48a467bd-18d5-4c16-a816-485dfb90a589 (at 0@lo) reconnecting [ 2004.339211] Lustre: 92575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2004.362502] LustreError: 93291:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2004.378050] LustreError: 93291:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2004.401656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2004.499126] Lustre: lustre-MDT0000: new disk, initializing [ 2004.628677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.654149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2004.669612] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2005.741228] Lustre: 95139:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000843dea94 x1875232178615936/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788360980 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2008.844532] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2010.848684] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2014.696832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2014.712809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2014.909877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2040.288730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2040.301100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.324311] Lustre: Skipped 1 previous similar message [ 2040.331601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.336053] Lustre: Skipped 1 previous similar message [ 2043.238504] Lustre: server umount lustre-OST0000 complete [ 2046.433713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.444579] Lustre: Skipped 1 previous similar message [ 2046.463249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2046.470455] Lustre: Skipped 4 previous similar messages [ 2051.555161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2051.566842] Lustre: Skipped 1 previous similar message [ 2052.407575] Lustre: server umount lustre-MDT0000 complete [ 2062.504839] Lustre: server umount lustre-MDT0001 complete [ 2065.684337] Lustre: server umount MGS complete [ 2076.009842] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:57:13 (1788361033) [ 2088.939882] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2091.082717] Key type lgssc unregistered [ 2091.362581] LNet: 97636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.386285] LNetError: 97636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.398684] LNet: Removed LNI 192.168.203.158@tcp [ 2092.160104] Key type .llcrypt unregistered [ 2092.161976] Key type ._llcrypt unregistered [ 2105.779487] Key type ._llcrypt registered [ 2105.781937] Key type .llcrypt registered [ 2105.873895] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2106.622805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2106.745297] alg: No test for adler32 (adler32-zlib) [ 2107.785309] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 2108.020889] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2109.687133] Key type lgssc registered [ 2110.661462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2117.332526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2117.352328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.678344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.092628] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2128.827156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.865976] LustreError: 98837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.878064] LustreError: 98837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2129.073837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2132.541330] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2138.460554] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2142.792716] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2147.955128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2148.230855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.867842] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2158.052306] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2163.631823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2168.806088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2168.806879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.810409] LustreError: Skipped 1 previous similar message [ 2168.828964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2168.836401] Lustre: Skipped 1 previous similar message [ 2170.523857] Lustre: server umount lustre-OST0000 complete [ 2173.920307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.924696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.931675] Lustre: Skipped 2 previous similar messages [ 2173.940459] Lustre: Skipped 1 previous similar message [ 2179.051734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2179.059110] Lustre: Skipped 1 previous similar message [ 2179.510626] Lustre: server umount lustre-MDT0000 complete [ 2182.447239] LustreError: 98816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361141 with bad export cookie 13395811790090856156 [ 2182.450287] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2182.461221] LustreError: 98816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2182.812373] Lustre: server umount lustre-MDT0001 complete [ 2191.716730] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2194.298893] Key type lgssc unregistered [ 2194.555649] LNet: 102367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.566424] LNetError: 102367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.623477] LNet: Removed LNI 192.168.203.158@tcp [ 2196.365951] Key type .llcrypt unregistered [ 2196.369554] Key type ._llcrypt unregistered [ 2210.912151] Key type ._llcrypt registered [ 2210.914214] Key type .llcrypt registered [ 2211.037601] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2212.038745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.081796] alg: No test for adler32 (adler32-zlib) [ 2213.126775] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 2213.355339] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2215.080433] Key type lgssc registered [ 2216.226654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2221.281700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2221.291350] LustreError: 103439:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2221.363684] LustreError: 103452:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2221.368798] LustreError: 103439:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4930 [ 2221.378024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.378605] LustreError: 103439:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2221.378625] LustreError: 103439:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2222.891984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.367844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.617102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.583202] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2238.767701] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing lsmod [ 2241.272788] LustreError: 104657:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2241.275286] LustreError: 104658:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2241.281168] LustreError: 104657:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2241.288038] LustreError: 104657:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2241.292248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2241.295067] LustreError: 104657:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2241.658131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.088399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2245.095294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.113690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.138517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.145308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.155599] Lustre: Skipped 1 previous similar message [ 2247.170735] Lustre: Skipped 1 previous similar message [ 2252.262117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.266386] Lustre: Skipped 3 previous similar messages [ 2254.320043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.332577] Lustre: Skipped 2 previous similar messages [ 2258.403500] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2258.556175] Lustre: server umount lustre-MDT0000 complete [ 2259.436480] LustreError: 103491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.459621] LustreError: 103491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2261.769429] LustreError: 103468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361220 with bad export cookie 15767290581102629834 [ 2261.770873] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2261.790808] LustreError: 103468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2262.559316] LustreError: 103921:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000004a75f32 x1875232604312576/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2262.717113] Lustre: server umount lustre-MDT0001 complete [ 2275.972480] Lustre: server umount lustre-OST0000 complete [ 2282.360231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.752210] LustreError: 105725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.837623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.983867] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2288.097631] LustreError: 105726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.567593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.832288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2294.563848] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2299.236580] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2303.567276] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2308.526970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2308.786971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.371757] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2318.687887] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2327.210997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2337.760141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2337.764664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.768070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2337.780693] Lustre: Skipped 1 previous similar message [ 2339.327159] Lustre: server umount lustre-OST0000 complete [ 2342.882587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2342.890790] LustreError: Skipped 1 previous similar message [ 2342.893120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.917834] Lustre: Skipped 1 previous similar message [ 2348.004270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.009528] Lustre: Skipped 4 previous similar messages [ 2348.301104] Lustre: server umount lustre-MDT0000 complete [ 2350.749155] LustreError: 108536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361309 with bad export cookie 15767290581102630527 [ 2350.750675] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2350.756115] LustreError: 108536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2350.987465] Lustre: server umount lustre-MDT0001 complete [ 2357.446774] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2359.334192] Key type lgssc unregistered [ 2359.599509] LNet: 109288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2359.607157] LNetError: 109288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.617511] LNet: Removed LNI 192.168.203.158@tcp [ 2360.200727] Key type .llcrypt unregistered [ 2360.205518] Key type ._llcrypt unregistered [ 2376.080893] Key type ._llcrypt registered [ 2376.082694] Key type .llcrypt registered [ 2376.155477] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2376.833902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2376.859257] alg: No test for adler32 (adler32-zlib) [ 2377.817374] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 2378.001102] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2379.655309] Key type lgssc registered [ 2380.503587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2385.854912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.872980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.209835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.131328] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2396.235546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.488367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2399.513384] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2404.426089] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2408.131532] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2412.448386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2412.634068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.677203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2416.353188] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2421.485446] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2426.304925] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2428.368292] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2432.483714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2432.487619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.491600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2432.503054] Lustre: Skipped 1 previous similar message [ 2436.886544] Lustre: server umount lustre-OST0000 complete [ 2442.721327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2442.724715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.729085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2442.737525] Lustre: Skipped 1 previous similar message [ 2442.743090] Lustre: Skipped 1 previous similar message [ 2447.481746] Lustre: server umount lustre-MDT0000 complete [ 2447.841454] LustreError: 110683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.856028] LustreError: 110683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2449.677606] LustreError: 110664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361408 with bad export cookie 7647755451928901343 [ 2449.679706] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.684238] LustreError: 110664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2449.883662] Lustre: server umount lustre-MDT0001 complete [ 2455.979577] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2457.735551] Key type lgssc unregistered [ 2457.924750] LNet: 114316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.931936] LNetError: 114316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.953508] LNet: Removed LNI 192.168.203.158@tcp [ 2458.543508] Key type .llcrypt unregistered [ 2458.545965] Key type ._llcrypt unregistered [ 2469.171673] Key type ._llcrypt registered [ 2469.174326] Key type .llcrypt registered [ 2469.264339] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2469.911567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2469.926366] alg: No test for adler32 (adler32-zlib) [ 2470.893020] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 2471.031703] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2472.663282] Key type lgssc registered [ 2473.389160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2478.190460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2478.210495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.528263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.286708] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2487.295393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.531392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2490.034158] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2494.248566] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2497.572180] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2501.889086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.132523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.180222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2505.495554] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2509.901226] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2519.522893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2519.529826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.543797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.592148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2522.597374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.598732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.611736] Lustre: Skipped 1 previous similar message [ 2524.737694] Lustre: server umount lustre-OST0000 complete [ 2527.713137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.727960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.732478] Lustre: Skipped 1 previous similar message [ 2532.832856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.837814] Lustre: Skipped 1 previous similar message [ 2533.459386] Lustre: server umount lustre-MDT0000 complete [ 2535.663655] LustreError: 118014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361494 with bad export cookie 7138461198991941362 [ 2535.668833] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2535.671719] LustreError: 118014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2535.910019] Lustre: server umount lustre-MDT0001 complete [ 2542.345733] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2543.892190] Key type lgssc unregistered [ 2544.037544] LNet: 119054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.042747] LNetError: 119054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2545.061628] LNet: Removed LNI 192.168.203.158@tcp [ 2545.638707] Key type .llcrypt unregistered [ 2545.640985] Key type ._llcrypt unregistered [ 2560.564679] Key type ._llcrypt registered [ 2560.566513] Key type .llcrypt registered [ 2560.632770] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2561.146186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2561.162661] alg: No test for adler32 (adler32-zlib) [ 2562.073703] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 2562.251528] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2563.871164] Key type lgssc registered [ 2564.646939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2569.510423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2569.530515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.836297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.438788] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2578.630746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.841637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2581.138860] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2585.363021] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2588.761258] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2592.716815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2592.936531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2596.016540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2596.561094] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2600.814878] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2608.725374] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2610.810785] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2610.817845] Lustre: Skipped 2 previous similar messages [ 2613.039841] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2613.043047] Lustre: Skipped 2 previous similar messages [ 2614.295506] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2614.311649] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2620.486206] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.58@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2620.490021] Lustre: Skipped 2 previous similar messages [ 2623.903137] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.58@tcp 192.168.0.[2,10]@tcp [ 2628.149819] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.58/24@tcp in log lustre-MDT0000 [ 2628.157970] Lustre: Skipped 2 previous similar messages [ 2630.410566] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.58/24@tcp [ 2630.414665] Lustre: Skipped 1 previous similar message [ 2634.211358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2634.218172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.231598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.235539] Lustre: Skipped 1 previous similar message [ 2639.331461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2639.335571] Lustre: Skipped 1 previous similar message [ 2640.040620] Lustre: server umount lustre-OST0000 complete [ 2644.449027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.451705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.460633] Lustre: Skipped 2 previous similar messages [ 2644.467142] Lustre: Skipped 1 previous similar message [ 2648.314548] Lustre: server umount lustre-MDT0000 complete [ 2649.570376] LustreError: 121932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.586764] LustreError: 121932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2650.415607] LustreError: 120432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361609 with bad export cookie 14566763123579218155 [ 2650.416845] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2650.425031] LustreError: 120432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2650.585557] Lustre: server umount lustre-MDT0001 complete [ 2655.885240] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2657.335637] Key type lgssc unregistered [ 2657.495670] LNet: 125284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2657.499620] LNetError: 125284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2657.509610] LNet: Removed LNI 192.168.203.158@tcp [ 2657.904835] Key type .llcrypt unregistered [ 2657.906964] Key type ._llcrypt unregistered [ 2672.087962] Key type ._llcrypt registered [ 2672.089522] Key type .llcrypt registered [ 2672.138174] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2672.638852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2672.689684] alg: No test for adler32 (adler32-zlib) [ 2673.591268] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 2673.718802] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2675.327225] Key type lgssc registered [ 2675.870031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2679.463691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2679.473251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.689863] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2680.702778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.706967] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.58/24@tcp [ 2682.648447] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2686.790729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.958871] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2686.969880] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.58/24@tcp [ 2686.984336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2688.875724] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2692.308522] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2694.850855] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2698.039688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.236224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.145563] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2704.814215] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2705.390509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2715.617699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2715.625439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.635963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.643557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2716.643951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.648136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.989949] Lustre: server umount lustre-OST0000 complete [ 2722.783874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2722.788086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.794715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2722.797960] Lustre: Skipped 1 previous similar message [ 2726.882691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.884357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.900074] Lustre: Skipped 1 previous similar message [ 2728.754323] Lustre: server umount lustre-MDT0000 complete [ 2730.574653] LustreError: 129655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361689 with bad export cookie 11019779859957331443 [ 2730.575561] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2730.582373] LustreError: 129655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2730.740614] Lustre: server umount lustre-MDT0001 complete [ 2735.320503] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2736.696513] Key type lgssc unregistered [ 2736.846405] LNet: 130406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2736.852256] LNetError: 130406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2736.864393] LNet: Removed LNI 192.168.203.158@tcp [ 2737.290943] Key type .llcrypt unregistered [ 2737.292763] Key type ._llcrypt unregistered [ 2750.336553] Key type ._llcrypt registered [ 2750.338712] Key type .llcrypt registered [ 2750.415673] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 2758.158608] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2758.565121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2758.573589] alg: No test for adler32 (adler32-zlib) [ 2759.480852] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 2759.588416] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2761.191164] Key type lgssc registered [ 2761.678777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2777.215418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2777.228735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.337935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2778.354943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2778.408811] Lustre: lustre-MDT0000: new disk, initializing [ 2778.448644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.461307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2780.560546] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2785.977760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.064921] Lustre: 133780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2786.116317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2786.119265] Lustre: Skipped 1 previous similar message [ 2786.165566] Lustre: lustre-MDT0001: new disk, initializing [ 2786.211885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.235925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2786.245290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2788.363390] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2793.183292] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2795.806413] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2799.019573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.169478] Lustre: lustre-OST0000: new disk, initializing [ 2799.171874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2799.177704] Lustre: 135748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2799.236614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.912484] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2805.766780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2805.773222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2805.821787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2806.535567] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2808.378215] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2808.465862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2810.169325] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2810.272665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2815.972134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.974777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2815.975858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2815.977978] Lustre: Skipped 1 previous similar message [ 2815.980351] LustreError: Skipped 1 previous similar message [ 2817.264203] Lustre: server umount lustre-OST0000 complete [ 2821.089101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.092190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.097537] Lustre: Skipped 1 previous similar message [ 2821.101015] Lustre: Skipped 1 previous similar message [ 2826.208087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.212573] Lustre: Skipped 1 previous similar message [ 2831.328202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.332878] Lustre: Skipped 1 previous similar message [ 2833.375962] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2833.461505] Lustre: server umount lustre-MDT0000 complete [ 2835.077698] LustreError: 133773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361793 with bad export cookie 16726909904057198564 [ 2835.080759] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.083120] LustreError: 133773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2835.212806] Lustre: server umount lustre-MDT0001 complete [ 2839.254338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.535032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.439368] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2844.640614] LustreError: 137967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.648541] LustreError: 137967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2845.054449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2845.216249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.902899] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2850.307224] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2852.746799] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2855.993547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.185132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.121899] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2862.707239] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2864.747885] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2864.878087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2867.050898] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2871.289443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 2875.859051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.943534] Lustre: lustre-OST0001: new disk, initializing [ 2875.946693] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2875.951992] Lustre: 141363:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2875.996716] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.968539] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2883.615895] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2885.142159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2885.149654] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2885.181163] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2885.813619] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2887.970932] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2888.088380] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2890.800296] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a08c4563800.ost_server_uuid 50 [ 2891.601169] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a08c4563800.ost_server_uuid in FULL state after 0 sec [ 2895.329779] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2895.335101] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.337055] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2895.342959] Lustre: Skipped 1 previous similar message [ 2901.200770] Lustre: server umount lustre-OST0001 complete [ 2905.571170] LustreError: 139785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.572244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.573113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2905.573119] Lustre: Skipped 3 previous similar messages [ 2905.580347] LustreError: 139785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2905.586747] Lustre: Skipped 1 previous similar message [ 2905.598939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2905.603360] LustreError: Skipped 1 previous similar message [ 2910.690439] LustreError: 142133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.700818] LustreError: 142133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2915.812051] LustreError: 139784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2915.820482] LustreError: 139784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2917.343115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2917.416410] Lustre: server umount lustre-OST0000 complete [ 2920.929093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.049892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2926.055439] Lustre: Skipped 8 previous similar messages [ 2933.215203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2933.293838] Lustre: server umount lustre-MDT0000 complete [ 2934.956161] LustreError: 138700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361893 with bad export cookie 16726909904057199446 [ 2934.959703] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2934.964845] LustreError: 138700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2935.143130] Lustre: server umount lustre-MDT0001 complete [ 2944.009293] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2945.378546] Key type lgssc unregistered [ 2945.519842] LNet: 144375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2945.523347] LNetError: 144375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2945.532939] LNet: Removed LNI 192.168.203.158@tcp [ 2945.950095] Key type .llcrypt unregistered [ 2945.953121] Key type ._llcrypt unregistered [ 2963.906851] Key type ._llcrypt registered [ 2963.908543] Key type .llcrypt registered [ 2963.954703] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 2971.439901] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2971.988058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2972.000808] alg: No test for adler32 (adler32-zlib) [ 2972.943519] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 2973.077158] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2974.679156] Key type lgssc registered [ 2975.175862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2990.728602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2990.750763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.863106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2991.883909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2991.928671] Lustre: lustre-MDT0000: new disk, initializing [ 2991.995108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.011303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2993.880640] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2998.866368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.931953] Lustre: 148501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2998.970852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2998.973801] Lustre: Skipped 1 previous similar message [ 2999.014448] Lustre: lustre-MDT0001: new disk, initializing [ 2999.064689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2999.088651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2999.094333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3000.948358] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3005.475404] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3007.973081] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3010.903793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3011.026325] Lustre: lustre-OST0000: new disk, initializing [ 3011.028742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3011.032400] Lustre: 150473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3011.070628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.752502] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3017.230229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3017.235647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3017.292196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3018.772242] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3021.228461] Lustre: Failing over lustre-OST0000 [ 3021.326595] Lustre: server umount lustre-OST0000 complete [ 3035.127815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.284867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.297710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3035.308257] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3035.702128] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3037.722508] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3038.367161] Lustre: 146728:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361981/real 1788361981] req@0000000045b456e6 x1875233400188416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788361997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3039.311333] Lustre: Failing over lustre-MDT0000 [ 3039.436834] Lustre: server umount lustre-MDT0000 complete [ 3040.224451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3040.229600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3040.236173] Lustre: Skipped 1 previous similar message [ 3043.807126] Lustre: 146730:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361986/real 1788361986] req@000000007495437b x1875233400189696/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788362002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3043.825091] Lustre: 146730:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3045.856028] LustreError: 148513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.863993] LustreError: 148513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3048.927326] Lustre: 146727:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788361991/real 1788361991] req@000000006b63bbc1 x1875233400190848/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788362007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3050.976192] LustreError: 149310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3050.986728] LustreError: 149310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3053.697204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.766050] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3053.774871] Lustre: 148505:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.158@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3053.880307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3053.908415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.938927] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3055.989413] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3056.179178] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3059.177649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3059.185636] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3059.219608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3064.292383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3064.296160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.299968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3064.310015] Lustre: Skipped 3 previous similar messages [ 3066.929931] Lustre: server umount lustre-OST0000 complete [ 3069.410401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.411246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3069.414652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.414659] Lustre: Skipped 1 previous similar message [ 3069.420725] Lustre: Skipped 1 previous similar message [ 3069.425531] LustreError: Skipped 1 previous similar message [ 3074.528179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3074.533281] Lustre: Skipped 2 previous similar messages [ 3075.175111] Lustre: server umount lustre-MDT0000 complete [ 3076.947155] LustreError: 150471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362035 with bad export cookie 4542127293235479969 [ 3076.950751] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3076.959306] LustreError: 150471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3077.128716] Lustre: server umount lustre-MDT0001 complete [ 3082.195984] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 3083.529953] Key type lgssc unregistered [ 3083.686474] LNet: 154117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3083.688837] LNetError: 154117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3083.701810] LNet: Removed LNI 192.168.203.158@tcp [ 3084.100635] Key type .llcrypt unregistered [ 3084.102113] Key type ._llcrypt unregistered [ 3107.100278] Key type ._llcrypt registered [ 3107.102853] Key type .llcrypt registered [ 3107.160455] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3107.756354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3107.766933] alg: No test for adler32 (adler32-zlib) [ 3108.661088] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 3108.795403] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 3110.399261] Key type lgssc registered [ 3110.910134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3114.751408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3114.763541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.973883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.773770] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3122.353856] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3124.601723] LustreError: 155661:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3124.605294] LustreError: 155661:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3124.609730] LustreError: 155661:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 3124.752448] Lustre: server umount lustre-MDT0000 complete [ 3131.236801] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 3135.079761] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3151.221399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.339914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3151.361352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3151.415373] Lustre: lustre-MDT0000: new disk, initializing [ 3151.459984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.471193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3153.329355] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3158.906707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.963820] Lustre: 158961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3159.004799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3159.008357] Lustre: Skipped 1 previous similar message [ 3159.053564] Lustre: lustre-MDT0001: new disk, initializing [ 3159.109228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3159.132621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3159.137744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3161.207274] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3166.093353] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3168.644863] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3171.697302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3171.823987] Lustre: lustre-OST0000: new disk, initializing [ 3171.826865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3171.830635] Lustre: 160928:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3171.869530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.223415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3174.228653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3174.285560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3174.435343] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3178.937440] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3182.187864] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3182.925702] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:15:41 (1788362141) [ 3287.007629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.007642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3287.007838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.023396] Lustre: Skipped 2 previous similar messages [ 3292.129437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3292.134832] Lustre: Skipped 2 previous similar messages [ 3297.247812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3297.252772] Lustre: Skipped 2 previous similar messages [ 3300.319158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3300.402454] Lustre: server umount lustre-MDT0000 complete [ 3301.919607] LustreError: 158952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362260 with bad export cookie 16740398872950332570 [ 3301.921148] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3301.928136] LustreError: 158952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3302.367506] LustreError: 158968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.367555] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.375633] LustreError: 158968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3302.384966] Lustre: Skipped 2 previous similar messages [ 3302.387160] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3306.399360] LustreError: 160936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3306.399462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3311.519633] LustreError: 158967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.191118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3316.309202] Lustre: server umount lustre-MDT0001 complete [ 3324.249470] Lustre: server umount lustre-OST0000 complete [ 3328.126844] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 3332.165161] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3347.008882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.115833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3347.131426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3347.173996] Lustre: lustre-MDT0000: new disk, initializing [ 3347.219467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.228278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3348.885771] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3354.378407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.433258] Lustre: 165561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3354.467316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3354.469825] Lustre: Skipped 1 previous similar message [ 3354.514439] Lustre: lustre-MDT0001: new disk, initializing [ 3354.573502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3354.580691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3356.720803] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3361.635960] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3364.081262] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3367.074512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.208189] Lustre: lustre-OST0000: new disk, initializing [ 3367.210836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3367.214067] Lustre: 167528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3367.254962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3367.258866] Lustre: Skipped 1 previous similar message [ 3367.561073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3367.566263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3367.624239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3370.062289] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3375.037222] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3377.058585] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3377.156660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3379.113133] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3379.235248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3383.267362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3383.268221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.271200] LustreError: Skipped 1 previous similar message [ 3383.279247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3383.285300] Lustre: Skipped 2 previous similar messages [ 3386.408361] Lustre: server umount lustre-OST0000 complete [ 3390.432648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3390.435091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.446640] Lustre: Skipped 1 previous similar message [ 3394.620430] Lustre: server umount lustre-MDT0000 complete [ 3396.493870] LustreError: 165553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362355 with bad export cookie 16740398872950491400 [ 3396.497735] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3396.500320] LustreError: 165553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3396.692271] Lustre: server umount lustre-MDT0001 complete [ 3400.314439] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:19:18 (1788362358) [ 3407.763262] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 3411.897114] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3428.431779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3428.535834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3428.563469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3428.615281] Lustre: lustre-MDT0000: new disk, initializing [ 3428.675153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.687114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3430.677647] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3436.392320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.463193] Lustre: 172274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3436.598581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3438.597555] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3443.327813] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3445.895576] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3448.957107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3449.095823] Lustre: 174252:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3450.045266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3450.092953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3451.507259] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3455.960529] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3460.576605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.578903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.580264] LustreError: Skipped 1 previous similar message [ 3460.586246] Lustre: Skipped 1 previous similar message [ 3460.588983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.594180] Lustre: Skipped 4 previous similar messages [ 3461.025156] Lustre: server umount lustre-OST0000 complete [ 3465.023833] Lustre: server umount lustre-MDT0000 complete [ 3465.701896] LustreError: 174924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3465.716659] LustreError: 174924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3466.966319] LustreError: 172267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362425 with bad export cookie 16740398872950492275 [ 3466.972664] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3466.974221] LustreError: 172267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3467.124666] Lustre: server umount lustre-MDT0001 complete [ 3471.178206] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:20:29 (1788362429) [ 3478.229723] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 3482.658741] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3499.249375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.366840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3499.393688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3499.397390] Lustre: Skipped 3 previous similar messages [ 3499.445408] Lustre: lustre-MDT0000: new disk, initializing [ 3499.448635] Lustre: Skipped 2 previous similar messages [ 3499.496947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.500017] Lustre: Skipped 2 previous similar messages [ 3499.508869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3499.513314] Lustre: Skipped 2 previous similar messages [ 3501.224683] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3506.737972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.804454] Lustre: 178566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3506.969854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3508.846481] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3513.611052] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3516.118965] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3519.447194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3519.604850] Lustre: 180543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3522.337181] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3526.588284] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3527.232094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3537.377561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3537.379755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.382320] LustreError: Skipped 1 previous similar message [ 3537.389097] Lustre: Skipped 2 previous similar messages [ 3537.391518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3537.395477] Lustre: Skipped 1 previous similar message [ 3540.326790] Lustre: server umount lustre-OST0000 complete [ 3556.319119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3556.402862] Lustre: server umount lustre-MDT0000 complete [ 3557.856498] LustreError: 179574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.868189] LustreError: 179574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3557.934439] LustreError: 178559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362516 with bad export cookie 16740398872950493367 [ 3557.935839] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.939468] LustreError: 178559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3558.097312] Lustre: server umount lustre-MDT0001 complete [ 3562.742077] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 3564.262919] Key type lgssc unregistered [ 3564.439649] LNet: 182399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3564.443885] LNetError: 182399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3564.453434] LNet: Removed LNI 192.168.203.158@tcp [ 3564.907188] Key type .llcrypt unregistered [ 3564.908975] Key type ._llcrypt unregistered [ 3577.510109] Key type ._llcrypt registered [ 3577.511521] Key type .llcrypt registered [ 3577.561892] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3578.129182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3578.149788] alg: No test for adler32 (adler32-zlib) [ 3579.068597] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 3579.209797] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 3580.815243] Key type lgssc registered [ 3581.398779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3585.056733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3585.069398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.344813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.803552] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3593.050954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.239981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3595.309559] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3599.053673] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3601.936787] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3605.440949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3605.691326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.639919] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3611.816553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3611.884237] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3634.657042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.657754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3634.657926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3634.662246] Lustre: Skipped 1 previous similar message [ 3639.777232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3639.781442] Lustre: Skipped 2 previous similar messages [ 3640.410902] Lustre: server umount lustre-OST0000 complete [ 3644.383844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3644.387066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.393360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3650.016339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3650.018907] Lustre: Skipped 2 previous similar messages [ 3655.138121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.143640] Lustre: Skipped 1 previous similar message [ 3656.160609] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3656.239891] Lustre: server umount lustre-MDT0000 complete [ 3657.674495] LustreError: 183772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362616 with bad export cookie 11548414743982725384 [ 3657.675297] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3657.678776] LustreError: 183772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3657.795944] Lustre: server umount lustre-MDT0001 complete [ 3662.071230] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 3663.258186] Key type lgssc unregistered [ 3663.402363] LNet: 187343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3663.407806] LNetError: 187343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3663.417497] LNet: Removed LNI 192.168.203.158@tcp [ 3663.794420] Key type .llcrypt unregistered [ 3663.797017] Key type ._llcrypt unregistered [ 3674.752579] Key type ._llcrypt registered [ 3674.754026] Key type .llcrypt registered [ 3674.801261] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3675.310738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3675.384703] alg: No test for adler32 (adler32-zlib) [ 3676.258712] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 3676.360348] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 3677.959275] Key type lgssc registered [ 3678.419903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3681.740720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3681.747651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.949292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.526837] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3688.345624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3688.512872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3690.181942] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3693.229598] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3695.541306] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3698.429790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3698.611826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3701.394881] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3705.019944] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3707.818483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3712.995597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.996530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3712.996996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3713.002444] Lustre: Skipped 1 previous similar message [ 3715.645742] Lustre: lustre-OST0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 3715.649575] Lustre: Skipped 1 previous similar message [ 3718.113143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.115611] Lustre: Skipped 1 previous similar message [ 3720.748575] Lustre: lustre-OST0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 3725.868843] Lustre: lustre-OST0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 3725.873989] Lustre: Skipped 2 previous similar messages [ 3727.327149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3727.397551] Lustre: server umount lustre-OST0000 complete [ 3732.587812] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a08c7435000.ost_server_uuid 50 [ 3734.066627] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3748.832782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.840679] Lustre: Skipped 1 previous similar message [ 3748.841301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3748.849744] Lustre: Skipped 1 previous similar message [ 3753.915909] Lustre: server umount lustre-MDT0000 complete [ 3753.956072] LustreError: 188737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.986551] LustreError: 188737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3756.875635] LustreError: 188718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362715 with bad export cookie 2612017319352385952 [ 3756.882979] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3756.888661] LustreError: 188718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3757.142510] Lustre: server umount lustre-MDT0001 complete [ 3765.154927] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:25:22 (1788362722) [ 3771.408259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.225175] LustreError: 192651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.310542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3777.362069] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3777.505049] LustreError: 192652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3782.628804] LustreError: 192651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3786.204100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.723737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3790.631555] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3797.082994] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3802.953082] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3810.001790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3810.302871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3816.420824] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3817.452108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3821.914741] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3827.106421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3827.231557] Lustre: lustre-OST0001: new disk, initializing [ 3827.240323] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3827.258027] Lustre: 195491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3827.370147] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3827.634807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3827.648509] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3827.757312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3831.441306] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3838.019828] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3842.016322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3842.022809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.034459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3843.048625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3845.850193] Lustre: server umount lustre-OST0000 complete [ 3848.164183] LustreError: 194472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.177706] LustreError: 194472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3849.710893] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3849.869862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3852.999197] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3853.171431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3858.413584] LustreError: 195532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.432155] LustreError: 195532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3868.644569] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3868.649039] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3868.651932] Lustre: Skipped 1 previous similar message [ 3871.573667] Lustre: server umount lustre-OST0001 complete [ 3878.882120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3878.887174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3878.892289] Lustre: Skipped 1 previous similar message [ 3878.895784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3878.907055] LustreError: Skipped 1 previous similar message [ 3878.918186] Lustre: Skipped 6 previous similar messages [ 3880.258379] Lustre: server umount lustre-MDT0000 complete [ 3883.065387] LustreError: 192632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362841 with bad export cookie 2612017319352387016 [ 3883.069441] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3883.077870] LustreError: 192632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3883.290289] Lustre: server umount lustre-MDT0001 complete [ 3897.914618] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:27:35 (1788362855) [ 3902.267227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3902.508099] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3902.561363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3902.718404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.859585] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3911.453475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.522209] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3911.556348] Lustre: 198924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3914.659145] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3919.986577] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3924.069522] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3929.233400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3929.382916] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3934.215823] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3937.874722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3939.163617] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3943.877325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3943.929311] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3947.432093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3948.444541] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3954.403320] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3957.738176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.740225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.745842] Lustre: Skipped 2 previous similar messages [ 3957.754949] Lustre: Skipped 1 previous similar message [ 3962.574531] Lustre: server umount lustre-OST0000 complete [ 3962.850949] LustreError: 200775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.877385] LustreError: 200775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3978.215401] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3978.218759] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.225597] LustreError: Skipped 1 previous similar message [ 3978.250637] Lustre: Skipped 1 previous similar message [ 3981.683873] Lustre: server umount lustre-OST0001 complete [ 3990.377788] Lustre: server umount lustre-MDT0000 complete [ 3993.139364] LustreError: 198916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362951 with bad export cookie 2612017319352388409 [ 3993.143220] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3993.147050] LustreError: 198916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3993.362602] Lustre: server umount lustre-MDT0001 complete [ 4008.429404] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:29:26 (1788362966) [ 4018.697635] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 4024.798678] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4046.441345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.591635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4046.612265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4046.700149] Lustre: lustre-MDT0000: new disk, initializing [ 4046.761617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.766694] Lustre: Skipped 3 previous similar messages [ 4046.777178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4049.201787] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4056.165714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.251394] Lustre: 207026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4056.295090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4056.297897] Lustre: Skipped 1 previous similar message [ 4056.340797] Lustre: lustre-MDT0001: new disk, initializing [ 4056.430080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4056.443127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4059.007661] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4065.397938] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4068.769957] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4072.711131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4072.890867] Lustre: lustre-OST0000: new disk, initializing [ 4072.894246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4072.900387] Lustre: 208994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4077.060750] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4082.700677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4082.709337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4082.766704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4083.247314] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4086.073082] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4086.232728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4088.662702] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4088.798740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4092.900777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4092.904311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4092.909749] LustreError: Skipped 2 previous similar messages [ 4092.910783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4092.910788] Lustre: Skipped 5 previous similar messages [ 4092.939145] Lustre: Skipped 3 previous similar messages [ 4104.161846] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4104.239686] Lustre: server umount lustre-OST0000 complete [ 4112.332189] Lustre: server umount lustre-MDT0000 complete [ 4113.376464] LustreError: 208266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.393503] LustreError: 208266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4114.331327] LustreError: 208993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363073 with bad export cookie 2612017319352390201 [ 4114.334214] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4114.342456] LustreError: 208993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4114.544758] Lustre: server umount lustre-MDT0001 complete [ 4119.610077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.368604] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4127.333361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.866567] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4134.202407] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4137.442234] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4141.343833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4145.011282] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4149.303283] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4151.912239] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4152.043864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4154.338760] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4154.455582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4157.921476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4157.925895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.929764] LustreError: Skipped 1 previous similar message [ 4157.939218] Lustre: Skipped 2 previous similar messages [ 4161.654605] Lustre: server umount lustre-OST0000 complete [ 4165.145354] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4167.687884] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4193.444522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.678084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.683725] Lustre: Skipped 5 previous similar messages [ 4197.424428] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4197.798410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4197.809789] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4197.813945] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4197.825429] Lustre: Skipped 1 previous similar message [ 4202.235693] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4213.912619] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 4218.115519] Lustre: server umount lustre-OST0000 complete [ 4230.674897] Lustre: server umount lustre-MDT0000 complete [ 4232.875667] LustreError: 215312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363191 with bad export cookie 2612017319352391083 [ 4232.879676] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4232.882945] LustreError: 215312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4242.516348] Lustre: server umount lustre-MDT0001 complete [ 4247.263986] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:33:25 (1788363205) [ 4250.667356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.887850] LustreError: 217190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.898170] LustreError: 217190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4253.168383] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4257.482622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.067871] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4263.968752] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4267.120977] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4270.994146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4274.477908] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4278.992558] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4281.384933] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4281.522807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4283.968817] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4284.080308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4287.632271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4287.746761] Lustre: lustre-OST0001: new disk, initializing [ 4287.749687] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4287.754948] Lustre: 220558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4290.866903] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4296.485438] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4297.768875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4297.776224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4297.828836] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4299.079868] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4299.221691] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4301.403778] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4301.537612] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4302.818212] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4302.820547] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4302.828478] LustreError: Skipped 1 previous similar message [ 4302.835105] Lustre: Skipped 2 previous similar messages [ 4304.584835] Lustre: server umount lustre-OST0001 complete [ 4307.899733] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4308.039374] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4310.383205] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4310.502771] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4335.608039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4339.183358] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4341.223727] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4341.228697] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4341.228839] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4343.311728] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4344.291619] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4351.440420] Lustre: server umount lustre-OST0001 complete [ 4366.827099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4366.830662] Lustre: Skipped 20 previous similar messages [ 4371.409205] Lustre: server umount lustre-OST0000 complete [ 4379.760360] Lustre: server umount lustre-MDT0000 complete [ 4381.604522] LustreError: 217171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363340 with bad export cookie 2612017319352392049 [ 4381.607452] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4381.612054] LustreError: 217171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4387.857840] Lustre: server umount lustre-MDT0001 complete [ 4399.864809] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:35:57 (1788363357) [ 4403.022878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4403.204149] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4403.228813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4405.460292] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4410.183880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.239033] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4410.254712] Lustre: 225792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4412.638281] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4416.244762] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4418.963805] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4422.333362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4422.436627] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4425.203419] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4428.647887] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4433.987672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4434.034525] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4437.028728] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4440.836821] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4441.660992] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4443.881315] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4443.993911] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4446.075879] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4446.202387] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4449.056944] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a08c4494800.ost_server_uuid 50 [ 4449.898672] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a08c4494800.ost_server_uuid in FULL state after 0 sec [ 4450.412596] Lustre: Permanently deactivating lustre-OST0001 [ 4450.422213] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4450.606553] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4451.796259] Lustre: Permanently reactivating lustre-OST0001 [ 4459.338447] Lustre: server umount lustre-OST0001 complete [ 4460.026129] LustreError: 227634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4460.033164] LustreError: 227634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 4475.359083] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4475.443283] Lustre: server umount lustre-OST0000 complete [ 4480.479129] LustreError: 230435:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 4487.702218] Lustre: server umount lustre-MDT0000 complete [ 4489.556585] LustreError: 225830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363448 with bad export cookie 2612017319352393323 [ 4489.559135] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4489.566569] LustreError: 225830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4489.736251] Lustre: server umount lustre-MDT0001 complete [ 4501.006071] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:37:39 (1788363459) [ 4504.490888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4504.646406] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4504.667236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4504.677777] Lustre: Skipped 5 previous similar messages [ 4504.779754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4504.782772] Lustre: Skipped 9 previous similar messages [ 4506.998792] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4511.554733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.632795] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4511.662821] Lustre: 232495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4514.274312] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4518.186213] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4521.258057] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4524.847271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4524.985899] Lustre: Permanently deactivating lustre-OST0000 [ 4524.998805] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4528.515335] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4532.998777] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4534.788434] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4534.795099] Lustre: Skipped 1 previous similar message [ 4534.798703] LustreError: 235144:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 4534.808047] LustreError: 235144:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 4536.747216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4540.154859] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4544.072257] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4544.187719] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4545.085889] Lustre: Permanently reactivating lustre-OST0000 [ 4545.095477] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4545.100276] Lustre: Skipped 2 previous similar messages [ 4553.230073] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4553.234626] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4553.243143] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4553.246065] Lustre: Skipped 1 previous similar message [ 4553.265297] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4553.284546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4555.747864] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4555.753431] LustreError: Skipped 3 previous similar messages [ 4559.335269] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4559.346862] Lustre: Skipped 14 previous similar messages [ 4560.674785] Lustre: server umount lustre-OST0001 complete [ 4577.247236] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4577.343363] Lustre: server umount lustre-OST0000 complete [ 4585.660167] Lustre: server umount lustre-MDT0000 complete [ 4587.707083] LustreError: 232486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363546 with bad export cookie 2612017319352395066 [ 4587.711049] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4587.716580] LustreError: 232486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4587.902883] Lustre: server umount lustre-MDT0001 complete [ 4593.167729] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 4594.521794] Key type lgssc unregistered [ 4594.671638] LNet: 237349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4594.680974] LNetError: 237349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4594.690704] LNet: Removed LNI 192.168.203.158@tcp [ 4595.139200] Key type .llcrypt unregistered [ 4595.144540] Key type ._llcrypt unregistered [ 4606.507039] Key type ._llcrypt registered [ 4606.508882] Key type .llcrypt registered [ 4606.569128] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4607.021441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4607.053791] alg: No test for adler32 (adler32-zlib) [ 4607.956418] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 4608.063992] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 4609.663239] Key type lgssc registered [ 4610.190993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4615.409738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4615.422497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.650098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4618.603368] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4622.943462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.014321] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4623.037633] Lustre: 238784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4623.046190] Lustre: Permanently deactivating lustre-MDT0001 [ 4623.056749] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4623.212452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4624.113038] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4624.118802] LustreError: 238826:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4624.122641] LustreError: 238826:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4624.126848] LustreError: 238826:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 4624.946396] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4628.267293] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4630.778884] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4634.309938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4634.510738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4637.720221] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4641.391479] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4641.702238] LustreError: 240632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4641.708432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4641.711179] LustreError: 240632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4644.765243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4644.879360] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4647.978305] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4649.012780] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4649.018402] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4651.658550] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4653.531720] Lustre: Permanently reactivating lustre-MDT0001 [ 4653.541295] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4653.544227] Lustre: Skipped 2 previous similar messages [ 4657.728495] Lustre: lustre-MDT0001: Client e3c93050-3744-4191-891b-a6687529a210 (at 192.168.203.58@tcp) reconnecting [ 4660.446736] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4661.151311] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4663.956214] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4664.056630] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4666.079425] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4666.181407] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4667.405564] Lustre: Permanently deactivating lustre-MDT0001 [ 4667.412819] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4667.416154] Lustre: Skipped 2 previous similar messages [ 4674.592475] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4684.767926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.775562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4684.778975] Lustre: Skipped 2 previous similar messages [ 4684.788225] Lustre: Skipped 3 previous similar messages [ 4686.396052] Lustre: server umount lustre-MDT0000 complete [ 4688.182960] LustreError: 242340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363647 with bad export cookie 1530430261989218787 [ 4688.184868] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4688.189534] LustreError: 242340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4688.358581] Lustre: server umount lustre-MDT0001 complete [ 4696.212644] Lustre: server umount lustre-OST0000 complete [ 4701.114910] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:40:59 (1788363659) [ 4706.137170] Lustre: server umount lustre-OST0001 complete [ 4708.920705] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 4712.708247] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4727.534551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.635457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4727.640502] Lustre: Skipped 2 previous similar messages [ 4727.652122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4727.694663] Lustre: lustre-MDT0000: new disk, initializing [ 4727.723312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4727.731205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4729.403632] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4734.365479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.411721] Lustre: 247148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4734.442185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4734.445123] Lustre: Skipped 1 previous similar message [ 4734.484833] Lustre: lustre-MDT0001: new disk, initializing [ 4734.536968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4734.545125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4736.164709] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4740.561268] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4743.200626] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4746.337460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4746.463478] Lustre: lustre-OST0000: new disk, initializing [ 4746.465993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4746.470565] Lustre: 249115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4746.511586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.514672] Lustre: Skipped 1 previous similar message [ 4749.164085] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4753.600564] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4753.919409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4753.926795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4753.970325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4759.733425] LustreError: 247154:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4764.437066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4764.531663] Lustre: lustre-OST0001: new disk, initializing [ 4764.533962] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4764.537719] Lustre: 250228:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4764.586131] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4767.056656] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4769.824989] LustreError: 247154:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4769.848803] LustreError: 249827:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4770.399512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4770.415421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4770.464038] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4771.977399] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4773.667284] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 4779.855782] LustreError: 249827:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4781.041414] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4781.048411] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.054925] Lustre: Skipped 1 previous similar message [ 4781.058702] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4782.650350] Lustre: lustre-OST0001: Not available for connect from 192.168.203.58@tcp (stopping) [ 4782.653563] Lustre: Skipped 1 previous similar message [ 4786.147388] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4786.150067] Lustre: Skipped 1 previous similar message [ 4791.268053] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4791.270338] Lustre: Skipped 2 previous similar messages [ 4794.847505] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4794.889935] Lustre: server umount lustre-OST0001 complete [ 4796.390370] LustreError: 249100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4796.396989] LustreError: 249100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4801.507915] LustreError: 249832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4801.515283] LustreError: 249832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4806.625416] LustreError: 249101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4806.633423] LustreError: 249101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4810.208930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4810.211765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4810.215611] LustreError: Skipped 2 previous similar messages [ 4810.221717] Lustre: Skipped 1 previous similar message [ 4810.224797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4810.229358] Lustre: Skipped 2 previous similar messages [ 4811.746236] LustreError: 249832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4811.753139] LustreError: 249832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4813.339658] Lustre: server umount lustre-OST0000 complete [ 4816.352075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4816.355041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4827.105240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4827.108084] Lustre: Skipped 7 previous similar messages [ 4829.151166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4829.220709] Lustre: server umount lustre-MDT0000 complete [ 4830.607181] LustreError: 247139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363789 with bad export cookie 1530430261989221377 [ 4830.610818] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4830.612932] LustreError: 247139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4836.801831] Lustre: server umount lustre-MDT0001 complete [ 4840.860016] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 4841.977372] Key type lgssc unregistered [ 4842.107362] LNet: 252260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4842.110641] LNetError: 252260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4842.121268] LNet: Removed LNI 192.168.203.158@tcp [ 4842.451749] Key type .llcrypt unregistered [ 4842.453387] Key type ._llcrypt unregistered [ 4857.833047] Key type ._llcrypt registered [ 4857.834365] Key type .llcrypt registered [ 4857.880213] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4858.354522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4858.374063] alg: No test for adler32 (adler32-zlib) [ 4859.255598] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 4859.355326] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 4860.935197] Key type lgssc registered [ 4861.405443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4864.287801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4864.297125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.399694] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4865.407743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4865.468654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4866.941203] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4870.068943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.138022] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4870.156339] Lustre: 254444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4870.266642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4871.741126] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4874.482619] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4876.590536] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4878.904521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4878.977804] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4879.049080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.075214] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4883.624862] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4883.987825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4930.016702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4930.017965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.020327] LustreError: Skipped 1 previous similar message [ 4930.025762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4930.029616] Lustre: Skipped 1 previous similar message [ 4934.493253] Lustre: server umount lustre-OST0000 complete [ 4936.051531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4940.401992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4940.515060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.534068] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4945.117369] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4949.735229] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.743215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.745578] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4949.753523] Lustre: Skipped 1 previous similar message [ 4980.707032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4980.708715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.717387] Lustre: Skipped 1 previous similar message [ 4980.721177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4980.726824] Lustre: Skipped 1 previous similar message [ 4984.804518] Lustre: server umount lustre-OST0000 complete [ 4988.384105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4988.387678] LustreError: Skipped 1 previous similar message [ 4988.389241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4988.394556] Lustre: Skipped 1 previous similar message [ 4988.399031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4990.944959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.945921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4990.953874] Lustre: Skipped 1 previous similar message [ 4992.685548] Lustre: server umount lustre-MDT0000 complete [ 4994.220322] LustreError: 254435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363953 with bad export cookie 16639758575395546336 [ 4994.221628] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4994.225683] LustreError: 254435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4994.384141] Lustre: server umount lustre-MDT0001 complete [ 4998.809882] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5000.057645] Key type lgssc unregistered [ 5000.206772] LNet: 259859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5000.212505] LNetError: 259859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5000.225542] LNet: Removed LNI 192.168.203.158@tcp [ 5000.582777] Key type .llcrypt unregistered [ 5000.584145] Key type ._llcrypt unregistered [ 5011.404540] Key type ._llcrypt registered [ 5011.406470] Key type .llcrypt registered [ 5011.458879] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5011.833738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5011.867200] alg: No test for adler32 (adler32-zlib) [ 5012.772331] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5012.883121] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5014.487216] Key type lgssc registered [ 5015.034149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5018.341553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5018.356775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.564482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5021.295671] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5024.846418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5025.016793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5026.618385] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5029.528401] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5031.755815] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5034.385561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5034.525214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.717451] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5039.344060] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5044.771295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5060.068126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5060.069442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5060.070912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5060.073849] Lustre: Skipped 1 previous similar message [ 5060.077224] LustreError: Skipped 1 previous similar message [ 5063.385643] Lustre: server umount lustre-OST0000 complete [ 5065.185543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.189817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5065.195358] Lustre: Skipped 1 previous similar message [ 5065.197995] Lustre: Skipped 1 previous similar message [ 5070.304594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5070.308135] Lustre: Skipped 2 previous similar messages [ 5070.938415] Lustre: server umount lustre-MDT0000 complete [ 5072.236171] LustreError: 261233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364031 with bad export cookie 8007892238506167292 [ 5072.239559] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5072.242694] LustreError: 261233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5072.358221] Lustre: server umount lustre-MDT0001 complete [ 5076.055655] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5077.065265] Key type lgssc unregistered [ 5077.192340] LNet: 265574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5077.196374] LNetError: 265574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5077.209395] LNet: Removed LNI 192.168.203.158@tcp [ 5077.537626] Key type .llcrypt unregistered [ 5077.539236] Key type ._llcrypt unregistered [ 5084.859526] Key type ._llcrypt registered [ 5084.861184] Key type .llcrypt registered [ 5084.908480] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5085.372808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5085.379753] alg: No test for adler32 (adler32-zlib) [ 5086.268744] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5086.373164] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5087.951196] Key type lgssc registered [ 5088.331140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5091.216251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5091.227041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.388377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5093.838383] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5096.756046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5096.885531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5098.356334] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5100.877563] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5102.818760] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5105.137986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5105.262080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5107.073883] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5109.630054] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5117.540965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5127.136014] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5127.138882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.144210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5128.161732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.161969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5128.162161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5129.743031] Lustre: server umount lustre-OST0000 complete [ 5132.767881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5132.770711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.774781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5132.777260] Lustre: Skipped 1 previous similar message [ 5137.178686] Lustre: server umount lustre-MDT0000 complete [ 5138.401457] LustreError: 266786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.409054] LustreError: 266786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5138.505233] LustreError: 266767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364097 with bad export cookie 15693919648040016440 [ 5138.508619] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5138.509038] LustreError: 266767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5138.631341] Lustre: server umount lustre-MDT0001 complete [ 5142.148779] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5143.124430] Key type lgssc unregistered [ 5143.248359] LNet: 270482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5143.251098] LNetError: 270482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5143.259298] LNet: Removed LNI 192.168.203.158@tcp [ 5143.585689] Key type .llcrypt unregistered [ 5143.586709] Key type ._llcrypt unregistered [ 5152.755107] Key type ._llcrypt registered [ 5152.756400] Key type .llcrypt registered [ 5152.804140] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5153.178245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5153.186246] alg: No test for adler32 (adler32-zlib) [ 5154.066887] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5154.184589] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5155.775205] Key type lgssc registered [ 5156.242541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5159.017229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5159.025087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.172595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5161.536374] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5164.305057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.426986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5165.759513] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5168.187882] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5170.138976] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5172.405686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5172.526057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5174.570893] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5177.206586] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5185.829899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5195.232355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5195.234495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.238390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5196.260099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5196.260238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5196.261745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.042860] Lustre: server umount lustre-OST0000 complete [ 5205.471720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5205.474445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5205.479684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5205.482207] Lustre: Skipped 1 previous similar message [ 5208.414239] Lustre: server umount lustre-MDT0000 complete [ 5209.702102] LustreError: 271856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364168 with bad export cookie 15138618986733028601 [ 5209.705535] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.707357] LustreError: 271856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5209.817667] Lustre: server umount lustre-MDT0001 complete [ 5213.433731] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5214.458532] Key type lgssc unregistered [ 5214.580355] LNet: 276196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5214.583488] LNetError: 276196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5214.592292] LNet: Removed LNI 192.168.203.158@tcp [ 5214.903114] Key type .llcrypt unregistered [ 5214.904195] Key type ._llcrypt unregistered [ 5221.998579] Key type ._llcrypt registered [ 5221.999740] Key type .llcrypt registered [ 5222.039156] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5222.477939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5222.493741] alg: No test for adler32 (adler32-zlib) [ 5223.349094] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5223.440636] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5225.023168] Key type lgssc registered [ 5225.368744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5227.829947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5227.836702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.976151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5230.330576] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5233.367026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.521280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5234.990987] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5237.371531] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5239.066229] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5241.171742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5241.276995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.086518] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5245.454653] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5253.605977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5263.841890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.843077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5263.846827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5263.850989] Lustre: Skipped 1 previous similar message [ 5265.424807] Lustre: server umount lustre-OST0000 complete [ 5268.960553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5268.961416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5268.964733] Lustre: Skipped 2 previous similar messages [ 5274.080164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5274.082261] Lustre: Skipped 2 previous similar messages [ 5279.200044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.201739] Lustre: Skipped 1 previous similar message [ 5281.247135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5281.304438] Lustre: server umount lustre-MDT0000 complete [ 5282.515249] LustreError: 277374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364241 with bad export cookie 2451343292435391370 [ 5282.517495] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5282.518403] LustreError: 277374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5282.641262] Lustre: server umount lustre-MDT0001 complete [ 5286.168739] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5287.285261] Key type lgssc unregistered [ 5287.414349] LNet: 281127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5287.418048] LNetError: 281127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5287.425280] LNet: Removed LNI 192.168.203.158@tcp [ 5287.756660] Key type .llcrypt unregistered [ 5287.758228] Key type ._llcrypt unregistered [ 5291.368679] Key type ._llcrypt registered [ 5291.369774] Key type .llcrypt registered [ 5291.404968] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5296.529780] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 5302.233781] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5302.563790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5302.587706] alg: No test for adler32 (adler32-zlib) [ 5303.436102] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5303.541933] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5305.135122] Key type lgssc registered [ 5305.479728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5315.966241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5315.971179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5317.046396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5317.055887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5317.090125] Lustre: lustre-MDT0000: new disk, initializing [ 5317.110479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5317.117263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5318.290048] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5322.154617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5322.192312] Lustre: 284651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5322.213156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5322.214735] Lustre: Skipped 1 previous similar message [ 5322.247918] Lustre: lustre-MDT0001: new disk, initializing [ 5322.276857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5322.289026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5322.292976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5323.519681] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5327.013252] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5328.784605] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5331.098822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5331.207071] Lustre: lustre-OST0000: new disk, initializing [ 5331.209688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5331.212894] Lustre: 286614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5331.242866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5332.921199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5332.925396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5332.956248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5333.043538] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5336.535392] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5337.848679] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5337.922717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5339.189240] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5339.252702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5343.201183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.202203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.203993] LustreError: Skipped 1 previous similar message [ 5343.208438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5343.212195] Lustre: Skipped 1 previous similar message [ 5345.928628] Lustre: server umount lustre-OST0000 complete [ 5347.807862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5347.810302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5347.814594] Lustre: Skipped 1 previous similar message [ 5347.816615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.248523] Lustre: server umount lustre-MDT0000 complete [ 5354.485114] LustreError: 284641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364313 with bad export cookie 8870651788136226207 [ 5354.488036] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5354.488673] LustreError: 284641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5354.609095] Lustre: server umount lustre-MDT0001 complete [ 5357.213800] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:51:55 (1788364315) [ 5359.272506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5359.417131] LustreError: 289022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.424401] LustreError: 289022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5359.443165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5360.649063] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5363.277758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.678760] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5367.091610] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5368.980775] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5371.180792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5371.307937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.310706] Lustre: Skipped 1 previous similar message [ 5373.209457] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5375.563792] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5389.793716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5389.794618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.795765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5389.795771] Lustre: Skipped 2 previous similar messages [ 5389.804214] Lustre: Skipped 2 previous similar messages [ 5394.912316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5394.914465] Lustre: Skipped 2 previous similar messages [ 5395.220481] Lustre: server umount lustre-OST0000 complete [ 5399.519834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5399.522459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.527020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5402.769796] Lustre: server umount lustre-MDT0000 complete [ 5404.055716] LustreError: 289758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364362 with bad export cookie 8870651788136227075 [ 5404.059286] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5404.060594] LustreError: 289758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5404.067623] LustreError: 289761:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5404.209816] Lustre: server umount lustre-MDT0001 complete [ 5408.062324] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5409.175066] Key type lgssc unregistered [ 5409.318606] LNet: 292576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5409.321881] LNetError: 292576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5409.331603] LNet: Removed LNI 192.168.203.158@tcp [ 5409.646056] Key type .llcrypt unregistered [ 5409.647452] Key type ._llcrypt unregistered [ 5418.291673] Key type ._llcrypt registered [ 5418.292879] Key type .llcrypt registered [ 5418.328773] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 5424.292595] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5424.613633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5424.674872] alg: No test for adler32 (adler32-zlib) [ 5425.525451] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5425.612386] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5427.191150] Key type lgssc registered [ 5427.537793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5441.590550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5441.594947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.666647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5442.676864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5442.712427] Lustre: lustre-MDT0000: new disk, initializing [ 5442.731381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5442.737759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5443.950653] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5447.919388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5447.966882] Lustre: 296372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5447.994730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5447.996476] Lustre: Skipped 1 previous similar message [ 5448.036597] Lustre: lustre-MDT0001: new disk, initializing [ 5448.070725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5448.084273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5448.087405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5449.406627] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5452.985127] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5454.624818] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5456.690368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5456.789165] Lustre: lustre-OST2710: new disk, initializing [ 5456.791128] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5456.793893] Lustre: 298339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5456.824214] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5458.623382] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5462.004534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5462.007533] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5462.036215] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5462.257939] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5464.553708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5464.614230] Lustre: lustre-OST03e8: new disk, initializing [ 5464.615842] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5464.617899] Lustre: 299400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5464.645527] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5466.474428] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5468.839114] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5470.212725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5470.215505] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5470.249715] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5471.432251] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5471.501886] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5472.938972] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5473.009368] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5474.397913] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5474.469757] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5475.840182] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5475.916548] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5478.879947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5478.882482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5478.887443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5480.416445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5480.419671] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5483.671190] Lustre: server umount lustre-MDT0000 complete [ 5484.891810] LustreError: 296364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364443 with bad export cookie 8680753759335014323 [ 5484.895260] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5484.896179] LustreError: 296364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5484.996969] Lustre: server umount lustre-MDT0001 complete [ 5496.448644] Lustre: server umount lustre-OST2710 complete [ 5507.933752] Lustre: server umount lustre-OST03e8 complete [ 5513.919489] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 5516.970334] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5528.494208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5528.573206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5528.582189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5528.614599] Lustre: lustre-MDT0000: new disk, initializing [ 5528.633097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.638166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5529.925540] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5533.867464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.905496] Lustre: 304639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5534.005071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5535.260238] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5538.819410] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5540.639908] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5542.798495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5542.889257] Lustre: lustre-OST0000: new disk, initializing [ 5542.891054] Lustre: Skipped 1 previous similar message [ 5542.892378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5542.894511] Lustre: Skipped 2 previous similar messages [ 5542.896322] Lustre: 306606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5542.920858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5542.922588] Lustre: Skipped 1 previous similar message [ 5544.503897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5544.507913] Lustre: Skipped 1 previous similar message [ 5544.510773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5544.535518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5544.793074] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5548.453905] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5549.961676] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5550.037822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5551.425493] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5551.495152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5554.658320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5554.658588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.661282] LustreError: Skipped 1 previous similar message [ 5554.665089] Lustre: Skipped 2 previous similar messages [ 5554.667219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5554.669593] Lustre: Skipped 4 previous similar messages [ 5558.220409] Lustre: server umount lustre-OST0000 complete [ 5559.776493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5559.776500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.776926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5559.789741] Lustre: Skipped 1 previous similar message [ 5564.895984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5564.898010] Lustre: Skipped 2 previous similar messages [ 5565.528372] Lustre: server umount lustre-MDT0000 complete [ 5566.771142] LustreError: 306605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364525 with bad export cookie 8680753759335015982 [ 5566.774148] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5566.774728] LustreError: 306605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5566.893306] Lustre: server umount lustre-MDT0001 complete [ 5569.815579] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:55:28 (1788364528) [ 5570.365425] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5570.963099] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:55:29 (1788364529) [ 5571.960659] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5580.681844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.771993] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5580.782924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5580.841867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5582.328731] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5585.163737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5585.190205] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5585.198640] Lustre: 310458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5586.599494] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5602.401876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.468095] Lustre: Denying initial registration attempt from nid 192.168.203.158@tcp, specified as failover [ 5602.471776] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5602.475439] LustreError: 311908:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5602.478336] LustreError: 311908:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5602.480949] LustreError: 311908:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5602.506636] Lustre: server umount lustre-OST0000 complete [ 5602.508718] LustreError: 311908:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5605.855756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5605.859592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.865958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5619.167149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5619.228640] Lustre: server umount lustre-MDT0000 complete [ 5620.495147] LustreError: 310449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364579 with bad export cookie 8680753759335016850 [ 5620.498063] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5620.498975] LustreError: 310449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5620.606873] Lustre: server umount lustre-MDT0001 complete [ 5624.057127] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5625.044292] Key type lgssc unregistered [ 5625.167363] LNet: 313027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5625.171071] LNetError: 313027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5625.177320] LNet: Removed LNI 192.168.203.158@tcp [ 5625.488978] Key type .llcrypt unregistered [ 5625.490099] Key type ._llcrypt unregistered [ 5629.248586] Key type ._llcrypt registered [ 5629.250376] Key type .llcrypt registered [ 5629.292416] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5629.719628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5629.774529] alg: No test for adler32 (adler32-zlib) [ 5630.628892] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5630.719644] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5632.303131] Key type lgssc registered [ 5632.663339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5656.312337] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5659.312909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5659.320143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.400601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5660.408242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5660.461461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5661.755108] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5664.382753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5664.410687] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5664.420688] Lustre: 315767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5664.511524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5665.743698] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5668.152722] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5669.869134] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5685.218414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5685.286498] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5685.348722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.196453] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5689.446770] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5690.146319] Lustre: server umount lustre-OST0000 complete [ 5692.400905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5692.401688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5692.404803] Lustre: Skipped 1 previous similar message [ 5692.409294] Lustre: Skipped 1 previous similar message [ 5697.486467] Lustre: server umount lustre-MDT0000 complete [ 5698.675150] LustreError: 315758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364657 with bad export cookie 6284157309091321945 [ 5698.678865] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.679983] LustreError: 315758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5698.794427] Lustre: server umount lustre-MDT0001 complete [ 5702.204439] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5703.191814] Key type lgssc unregistered [ 5703.317371] LNet: 319332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5703.319889] LNetError: 319332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5703.327320] LNet: Removed LNI 192.168.203.158@tcp [ 5703.617584] Key type .llcrypt unregistered [ 5703.618687] Key type ._llcrypt unregistered [ 5712.930710] Key type ._llcrypt registered [ 5712.932066] Key type .llcrypt registered [ 5712.970257] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5713.292928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5713.323341] alg: No test for adler32 (adler32-zlib) [ 5714.182784] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5714.274141] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5715.855099] Key type lgssc registered [ 5716.216833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5718.815319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5718.822198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5719.979283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5721.413500] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5724.354390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5724.480630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5725.861445] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5728.289490] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5730.151763] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5734.433736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5734.542531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.461255] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5738.851795] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5754.417864] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 5755.360649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5755.362036] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.362772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5755.368645] Lustre: Skipped 2 previous similar messages [ 5757.528197] Lustre: server umount lustre-MDT0000 complete [ 5758.700210] LustreError: 320704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364717 with bad export cookie 16729056269257859108 [ 5758.703622] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5758.704070] LustreError: 320704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5759.531748] LustreError: 323212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.539069] LustreError: 323212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5759.539329] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.58@tcp (stopping) [ 5759.544597] Lustre: Skipped 2 previous similar messages [ 5760.479855] LustreError: 320718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.480197] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.487711] LustreError: 323212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.488056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5763.495889] Lustre: Skipped 1 previous similar message [ 5768.608161] LustreError: 323212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.608278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5768.612324] LustreError: 323212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5768.613767] Lustre: Skipped 1 previous similar message [ 5773.279129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5773.356436] Lustre: server umount lustre-MDT0001 complete [ 5775.460070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5778.169747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5784.031479] LustreError: 320036:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000492d749f x1875236275380736/t0(0) o250->MGC192.168.203.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5784.131200] LustreError: 322553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5784.135884] LustreError: 322553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5784.151022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.167276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5785.322131] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5787.893063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5789.216594] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5791.355438] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5798.374143] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5800.001546] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 5801.788493] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5802.307951] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5804.511857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5804.515682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.522140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5804.524733] Lustre: Skipped 1 previous similar message [ 5808.612459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5808.616136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.685284] Lustre: server umount lustre-OST0000 complete [ 5813.728660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.729095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5817.040548] Lustre: server umount lustre-MDT0000 complete [ 5818.130264] LustreError: 320705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364776 with bad export cookie 16729056269257866227 [ 5818.133272] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5818.136170] LustreError: 320705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5818.243875] Lustre: server umount lustre-MDT0001 complete [ 5821.461571] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5822.390781] Key type lgssc unregistered [ 5822.509328] LNet: 327005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5822.512282] LNetError: 327005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5822.519238] LNet: Removed LNI 192.168.203.158@tcp [ 5822.789640] Key type .llcrypt unregistered [ 5822.791076] Key type ._llcrypt unregistered [ 5831.497476] Key type ._llcrypt registered [ 5831.498965] Key type .llcrypt registered [ 5831.537249] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5831.942137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5831.948186] alg: No test for adler32 (adler32-zlib) [ 5832.797258] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5832.883965] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5834.463166] Key type lgssc registered [ 5834.776055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5837.212606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5837.218023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5838.343374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5839.600633] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5842.208960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5842.324437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5843.567163] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5845.848333] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5847.640755] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5852.639788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5852.642284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.646185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5853.664307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5853.664838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.739219] Lustre: server umount lustre-MDT0000 complete [ 5855.879967] LustreError: 328378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364814 with bad export cookie 6249673368868016309 [ 5855.883250] LustreError: 328378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5855.883330] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5855.993983] Lustre: server umount lustre-MDT0001 complete [ 5860.622330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5860.710679] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5860.723367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5860.792026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5862.097236] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5864.870305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5864.902708] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5864.911612] Lustre: 330982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5865.005641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5866.295721] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5868.661824] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5870.474315] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5874.785580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5874.845226] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5874.848261] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5874.850787] LustreError: 332863:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5874.852801] LustreError: 332863:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5874.854540] LustreError: 332863:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5874.873207] Lustre: server umount lustre-OST0000 complete [ 5874.874396] LustreError: 332863:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5879.522350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.588819] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5879.655871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.568109] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5884.008588] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5885.970437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5886.556383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5886.588127] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5886.618347] Lustre: lustre-OST0001: new disk, initializing [ 5886.620207] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5886.623362] Lustre: 334401:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5888.536168] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5890.889971] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5891.887168] Lustre: server umount lustre-OST0001 complete [ 5895.662561] LustreError: 333330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.664944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.669535] LustreError: 333330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5895.677660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.679985] Lustre: Skipped 2 previous similar messages [ 5899.343404] Lustre: server umount lustre-OST0000 complete [ 5900.770540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.770902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5900.771693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.777565] Lustre: Skipped 2 previous similar messages [ 5902.744367] Lustre: server umount lustre-MDT0000 complete [ 5903.919981] LustreError: 330972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364862 with bad export cookie 6249673368868016848 [ 5903.922262] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5903.923229] LustreError: 330972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5904.035133] Lustre: server umount lustre-MDT0001 complete [ 5907.376643] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5908.431064] Key type lgssc unregistered [ 5908.562296] LNet: 336363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5908.565791] LNetError: 336363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5908.573335] LNet: Removed LNI 192.168.203.158@tcp [ 5908.882090] Key type .llcrypt unregistered [ 5908.883369] Key type ._llcrypt unregistered [ 5939.182387] Key type ._llcrypt registered [ 5939.184227] Key type .llcrypt registered [ 5939.222570] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 5945.312233] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5945.784469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5945.793709] alg: No test for adler32 (adler32-zlib) [ 5946.656716] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 5946.767780] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5948.359153] Key type lgssc registered [ 5948.754689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5958.850395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5958.855276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.938152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5959.948566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5959.987131] Lustre: lustre-MDT0000: new disk, initializing [ 5960.008678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5960.015537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5961.296078] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5965.390650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.425322] Lustre: 341263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5965.447499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5965.450302] Lustre: Skipped 1 previous similar message [ 5965.486764] Lustre: lustre-MDT0001: new disk, initializing [ 5965.539140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5965.552252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5965.555927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5966.869853] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5970.497658] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5972.192089] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5974.383382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5974.482324] Lustre: lustre-OST0000: new disk, initializing [ 5974.485072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5974.489404] Lustre: 343231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5974.524714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5976.311274] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5978.105791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5978.109185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5978.148797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5979.951409] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5981.275059] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5981.340866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5982.510518] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5982.577378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5983.202405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.203853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5983.206230] Lustre: Skipped 1 previous similar message [ 5983.209460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5985.237531] Lustre: server umount lustre-OST0000 complete [ 5988.321943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.322603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5988.325489] Lustre: Skipped 1 previous similar message [ 5988.329555] Lustre: Skipped 2 previous similar messages [ 5992.543197] Lustre: server umount lustre-MDT0000 complete [ 5993.778241] LustreError: 342058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364952 with bad export cookie 2973880945653069805 [ 5993.781680] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5993.781768] LustreError: 342058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5993.903274] Lustre: server umount lustre-MDT0001 complete [ 5996.820519] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:02:35 (1788364955) [ 5999.154595] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:02:37 (1788364957) [ 6001.667781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.795221] LustreError: 345984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.800656] LustreError: 345984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6001.817224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6003.002401] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6005.713263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.819872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.087382] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6009.387326] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6011.031319] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6013.052828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6014.864693] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6016.965888] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6030.654132] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6031.197258] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6031.740460] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6032.331887] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6032.960825] Lustre: Failing over lustre-MDT0000 [ 6033.076972] Lustre: server umount lustre-MDT0000 complete [ 6036.448212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6036.453040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.459601] LustreError: 345984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.555081] LustreError: 345981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.559721] LustreError: 345981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6042.080053] LustreError: 348336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.084773] LustreError: 348336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6046.066762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.109394] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6046.204659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.207854] Lustre: Skipped 1 previous similar message [ 6046.235232] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6047.477851] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6047.788099] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6051.300842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6051.307524] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6051.324225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6053.004888] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6053.477344] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6055.297674] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6055.909329] Lustre: Failing over lustre-MDT0000 [ 6056.050083] Lustre: server umount lustre-MDT0000 complete [ 6056.417702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.418510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6056.418852] LustreError: 345981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.418864] LustreError: 345981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6056.421673] Lustre: Skipped 4 previous similar messages [ 6066.656663] LustreError: 348336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.660934] LustreError: 348336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6068.924101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.975076] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6069.071372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.090432] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6070.464322] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6072.890147] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6074.340953] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6074.343016] Lustre: Skipped 2 previous similar messages [ 6074.350829] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6074.369434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6076.190800] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6076.713991] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6079.457373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6079.458154] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.458930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6079.467106] Lustre: Skipped 2 previous similar messages [ 6084.576784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6084.579616] Lustre: Skipped 4 previous similar messages [ 6085.078424] Lustre: server umount lustre-MDT0000 complete [ 6086.279213] LustreError: 345965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365045 with bad export cookie 2973880945653072458 [ 6086.282722] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6086.284337] LustreError: 345965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6086.393701] Lustre: server umount lustre-MDT0001 complete [ 6097.877483] Lustre: server umount lustre-OST0000 complete [ 6102.781806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6102.920321] LustreError: 352334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6102.926844] LustreError: 352334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6102.942485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6104.250819] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6106.969788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.452250] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6110.834439] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6112.562842] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6114.645895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6116.476473] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6118.766662] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6128.037373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6132.191834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6132.195340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.202285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6136.394159] Lustre: server umount lustre-OST0000 complete [ 6143.456132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6143.458357] Lustre: Skipped 6 previous similar messages [ 6143.770534] Lustre: server umount lustre-MDT0000 complete [ 6145.086247] LustreError: 352316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365103 with bad export cookie 2973880945653073200 [ 6145.089830] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6145.091596] LustreError: 352316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6145.206080] Lustre: server umount lustre-MDT0001 complete [ 6149.011394] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6150.010427] Key type lgssc unregistered [ 6150.146429] LNet: 355882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6150.149108] LNetError: 355882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6150.156348] LNet: Removed LNI 192.168.203.158@tcp [ 6150.470248] Key type .llcrypt unregistered [ 6150.471387] Key type ._llcrypt unregistered [ 6159.584711] Key type ._llcrypt registered [ 6159.585821] Key type .llcrypt registered [ 6159.627292] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6160.058671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6160.087618] alg: No test for adler32 (adler32-zlib) [ 6160.957409] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6161.057681] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6162.639114] Key type lgssc registered [ 6162.975963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6165.657380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6165.663974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6166.809344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6168.156814] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6170.926744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.052284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6172.389364] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6174.815963] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6176.681939] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6178.734808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6178.825987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.618825] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6182.819000] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6192.165438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6197.217820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6197.218760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6197.220991] Lustre: Skipped 2 previous similar messages [ 6197.223415] Lustre: Skipped 1 previous similar message [ 6202.337334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6202.339024] Lustre: Skipped 3 previous similar messages [ 6203.031424] Lustre: server umount lustre-MDT0000 complete [ 6204.223242] LustreError: 359216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365163 with bad export cookie 4490525004281284681 [ 6204.228379] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6204.346931] Lustre: server umount lustre-MDT0001 complete [ 6215.765650] Lustre: server umount lustre-OST0000 complete [ 6220.295924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6220.444285] LustreError: 361099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.452172] LustreError: 361099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6220.470898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6221.849329] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6224.777486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.895757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6226.143611] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6228.344229] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6230.041387] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6232.271336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6234.209348] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6236.408867] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6245.668730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6255.583986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6255.587608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.594377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6259.934375] Lustre: server umount lustre-MDT0000 complete [ 6261.191802] LustreError: 361080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365220 with bad export cookie 4490525004281285773 [ 6261.195887] LustreError: 361080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6261.196884] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6261.215988] LustreError: 361095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6261.216264] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.224052] Lustre: Skipped 2 previous similar messages [ 6261.225682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6261.227523] Lustre: Skipped 3 previous similar messages [ 6261.320876] Lustre: server umount lustre-MDT0001 complete [ 6268.643991] Lustre: server umount lustre-OST0000 complete [ 6273.691380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.849249] LustreError: 365027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6273.870574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6273.873251] Lustre: Skipped 1 previous similar message [ 6275.330645] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6278.236352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.876180] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6282.069865] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6283.760685] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6285.879858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6287.717183] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6289.939424] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6298.278259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6303.716728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.717809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6303.718769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6303.721099] Lustre: Skipped 1 previous similar message [ 6307.532831] Lustre: server umount lustre-OST0000 complete [ 6308.832182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.837959] Lustre: Skipped 1 previous similar message [ 6313.952015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.953753] Lustre: Skipped 4 previous similar messages [ 6323.167109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6323.226758] Lustre: server umount lustre-MDT0000 complete [ 6324.507145] LustreError: 365008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365283 with bad export cookie 4490525004281286788 [ 6324.510471] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6324.511288] LustreError: 365008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6324.637325] Lustre: server umount lustre-MDT0001 complete [ 6328.167750] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6329.203584] Key type lgssc unregistered [ 6329.333401] LNet: 368577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6329.335694] LNetError: 368577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6329.343303] LNet: Removed LNI 192.168.203.158@tcp [ 6329.658873] Key type .llcrypt unregistered [ 6329.659845] Key type ._llcrypt unregistered [ 6339.039990] Key type ._llcrypt registered [ 6339.042642] Key type .llcrypt registered [ 6339.087647] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6339.521363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6339.615940] alg: No test for adler32 (adler32-zlib) [ 6340.492874] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6340.603555] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6342.191122] Key type lgssc registered [ 6342.594480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6345.297418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6345.303646] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6345.307155] LustreError: 369996:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6345.318175] LustreError: 369996:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6345.322492] LustreError: 369996:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6345.326967] LustreError: 369996:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6345.332063] LustreError: 369996:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6347.858505] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6347.861207] LustreError: 370393:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6347.868825] LustreError: 370393:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6347.871305] LustreError: 370393:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6347.873465] LustreError: 370393:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6347.875943] LustreError: 370393:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6354.072701] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6355.062969] Key type lgssc unregistered [ 6355.184384] LNet: 371395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6355.186426] LNetError: 371395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6355.195255] LNet: Removed LNI 192.168.203.158@tcp [ 6355.505355] Key type .llcrypt unregistered [ 6355.506499] Key type ._llcrypt unregistered [ 6361.351930] Key type ._llcrypt registered [ 6361.353098] Key type .llcrypt registered [ 6361.395367] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 6367.700944] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6368.158797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6368.220888] alg: No test for adler32 (adler32-zlib) [ 6369.081093] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6369.170268] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6370.751165] Key type lgssc registered [ 6371.130605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6382.252707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6382.257986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6383.345555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6383.355187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6383.393367] Lustre: lustre-MDT0000: new disk, initializing [ 6383.414141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6383.420486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6384.799630] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6388.954206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.006160] Lustre: 374524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6389.033312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6389.036339] Lustre: Skipped 1 previous similar message [ 6389.076971] Lustre: lustre-MDT0001: new disk, initializing [ 6389.113427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6389.125062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6389.128165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6390.500595] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6394.114086] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6395.844563] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6398.035500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6398.144311] Lustre: lustre-OST0000: new disk, initializing [ 6398.147511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6398.150354] Lustre: 376488:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6398.178257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6400.063354] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6401.912405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6401.916091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6401.950267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6403.758170] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6405.094018] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6405.167096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6406.474645] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6406.550660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6412.257755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6412.258544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.261491] LustreError: Skipped 1 previous similar message [ 6412.267464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6412.270557] Lustre: Skipped 1 previous similar message [ 6413.264148] Lustre: server umount lustre-OST0000 complete [ 6414.815865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6414.818047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.821253] Lustre: Skipped 1 previous similar message [ 6414.822868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6417.376225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6417.376935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.639091] Lustre: server umount lustre-MDT0000 complete [ 6421.886207] LustreError: 374515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365380 with bad export cookie 8368745285559202807 [ 6421.889333] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.890091] LustreError: 374515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6422.010517] Lustre: server umount lustre-MDT0001 complete [ 6424.708133] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6425.230445] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:09:43 (1788365383) [ 6427.333059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.502853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6428.814231] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6431.526652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.643571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6432.978816] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6435.289695] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6437.072875] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6439.320103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6441.327351] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6443.699725] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6446.017796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6446.084081] Lustre: lustre-OST0001: new disk, initializing [ 6446.085701] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6446.088146] Lustre: 381780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6446.119040] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6446.121740] Lustre: Skipped 1 previous similar message [ 6448.040801] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6448.578404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6448.583355] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6448.618504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6451.740525] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6453.728092] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6453.728297] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.730746] LustreError: Skipped 1 previous similar message [ 6453.731258] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6453.736702] Lustre: Skipped 1 previous similar message [ 6453.738102] Lustre: Skipped 1 previous similar message [ 6457.830624] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6457.835627] Lustre: Skipped 2 previous similar messages [ 6459.643462] Lustre: server umount lustre-OST0001 complete [ 6462.947696] LustreError: 381155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.951822] LustreError: 381155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6468.067045] LustreError: 380767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6468.071987] LustreError: 380767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6472.159832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6472.162744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.166761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6472.169315] Lustre: Skipped 1 previous similar message [ 6473.186195] LustreError: 380768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.190326] LustreError: 380768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6477.687362] Lustre: server umount lustre-OST0000 complete [ 6482.913727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6482.919368] LustreError: Skipped 1 previous similar message [ 6482.929200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.950394] Lustre: Skipped 1 previous similar message [ 6486.535558] Lustre: server umount lustre-MDT0000 complete [ 6488.575115] LustreError: 378945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.639853] LustreError: 378945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6492.642961] LustreError: 378927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365451 with bad export cookie 8368745285559203675 [ 6492.648190] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6492.663537] LustreError: 378927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6499.485155] Lustre: server umount lustre-MDT0001 complete [ 6511.071917] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6512.547354] Key type lgssc unregistered [ 6512.697487] LNet: 383764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6512.701557] LNetError: 383764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6512.713674] LNet: Removed LNI 192.168.203.158@tcp [ 6513.077406] Key type .llcrypt unregistered [ 6513.078861] Key type ._llcrypt unregistered [ 6543.986897] Key type ._llcrypt registered [ 6543.991957] Key type .llcrypt registered [ 6544.042516] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6544.654632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6544.677651] alg: No test for adler32 (adler32-zlib) [ 6545.591110] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6545.722801] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6547.319523] Key type lgssc registered [ 6547.844852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6551.647292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6551.658805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.812974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6552.827177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6552.894796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6554.655375] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6558.355529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6558.423902] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6558.437653] Lustre: 386539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6558.553743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6560.218761] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6563.514811] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6566.028155] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6568.928193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6568.935434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.945088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6571.488166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6571.488758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.164931] Lustre: server umount lustre-MDT0000 complete [ 6574.762505] LustreError: 386575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365533 with bad export cookie 599900010341808337 [ 6574.765590] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6574.768428] LustreError: 386575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6574.906825] Lustre: server umount lustre-MDT0001 complete [ 6578.838918] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:12:16 (1788365536) [ 6581.672437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.947831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.705023] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6587.360437] LustreError: 389049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6587.368596] LustreError: 389049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6587.400710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.571044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6589.225348] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6592.260440] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6594.708193] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6597.741605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6597.842624] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6597.941535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6600.658248] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6604.066327] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6609.125185] Lustre: Permanently deactivating lustre-OST0000 [ 6609.134735] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6614.522949] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6615.407720] LustreError: 392034:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6615.412333] LustreError: 392034:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 6615.415721] LustreError: 392034:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6616.130854] LustreError: 392083:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 6616.135349] LustreError: 392083:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6623.079877] Lustre: server umount lustre-OST0000 complete [ 6624.279469] LustreError: 392332:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 6624.282686] LustreError: 392332:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6628.832336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6628.835354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6628.841461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6628.843820] Lustre: Skipped 1 previous similar message [ 6634.976248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.980627] Lustre: Skipped 3 previous similar messages [ 6635.010686] Lustre: server umount lustre-MDT0000 complete [ 6636.569285] LustreError: 389031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365595 with bad export cookie 599900010341808904 [ 6636.572675] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6636.573736] LustreError: 389031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6646.864989] Lustre: server umount lustre-MDT0001 complete [ 6651.217468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.034158] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6654.284429] LustreError: 393489:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6657.207855] Lustre: 393827:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.158@tcp [ 6658.241029] Lustre: server umount MGS complete [ 6663.291297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.494084] LustreError: 394553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6663.501977] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6663.509163] Lustre: Skipped 1 previous similar message [ 6663.542905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6665.286071] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6668.768771] LustreError: 394554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6669.149958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6669.289643] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6671.257208] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6674.621178] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6677.302791] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6678.250361] Lustre: Permanently reactivating lustre-OST0000 [ 6678.259707] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6678.263199] Lustre: Skipped 2 previous similar messages [ 6698.463122] LustreError: 396130:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6701.468082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6701.647994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.652143] Lustre: Skipped 1 previous similar message [ 6704.238563] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6707.607249] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6724.577856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6724.578412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6724.581473] LustreError: Skipped 1 previous similar message [ 6724.581865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6724.589555] Lustre: Skipped 2 previous similar messages [ 6726.329890] Lustre: server umount lustre-OST0000 complete [ 6729.698046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.705583] Lustre: Skipped 1 previous similar message [ 6734.183590] Lustre: server umount lustre-MDT0000 complete [ 6734.816116] LustreError: 394549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6734.821591] LustreError: 394549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6735.737518] LustreError: 395288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365694 with bad export cookie 599900010341809940 [ 6735.741391] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6735.744788] LustreError: 395288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6735.888389] Lustre: server umount lustre-MDT0001 complete [ 6740.653464] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6741.889857] Key type lgssc unregistered [ 6742.067151] LNet: 398934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6742.072743] LNetError: 398934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6742.085570] LNet: Removed LNI 192.168.203.158@tcp [ 6742.553494] Key type .llcrypt unregistered [ 6742.555443] Key type ._llcrypt unregistered [ 6750.372644] Key type ._llcrypt registered [ 6750.373885] Key type .llcrypt registered [ 6750.421316] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 6758.232718] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6758.712396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6758.761824] alg: No test for adler32 (adler32-zlib) [ 6759.668862] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6759.814042] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6761.423648] Key type lgssc registered [ 6762.083637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6780.260365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6780.270519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6781.404703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6781.439437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6781.512469] Lustre: lustre-MDT0000: new disk, initializing [ 6781.572048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.590976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6783.889332] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6789.006190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.067359] Lustre: 402065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6789.099474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6789.102636] Lustre: Skipped 1 previous similar message [ 6789.144433] Lustre: lustre-MDT0001: new disk, initializing [ 6789.199813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6789.219541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6789.224524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6790.948511] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6795.713551] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6798.296530] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6801.524859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6801.652728] Lustre: lustre-OST0000: new disk, initializing [ 6801.659541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6801.665389] Lustre: 404034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6801.710592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6804.455765] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6809.029136] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6810.870374] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6811.668752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6811.675733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6811.744639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6811.999391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6814.160453] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6814.257530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6816.738688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6816.744932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6816.751474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6816.754025] Lustre: Skipped 1 previous similar message [ 6821.345728] Lustre: server umount lustre-OST0000 complete [ 6824.928188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6824.930881] LustreError: Skipped 1 previous similar message [ 6824.932981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.938384] Lustre: Skipped 1 previous similar message [ 6824.941299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6826.976332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6826.977486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6829.322881] Lustre: server umount lustre-MDT0000 complete [ 6831.251949] LustreError: 402056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365790 with bad export cookie 11408891718083840081 [ 6831.255457] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6831.258445] LustreError: 402056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6831.414771] Lustre: server umount lustre-MDT0001 complete [ 6835.573072] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6836.355369] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:16:34 (1788365794) [ 6839.254315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.522883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6841.317517] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6844.791200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.895766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6844.898255] Lustre: Skipped 1 previous similar message [ 6844.943942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6846.593585] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6849.605314] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6852.112621] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6855.479883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6855.656910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.447786] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6862.045408] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6862.955918] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6870.181144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6875.107471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.109316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.114117] Lustre: Skipped 1 previous similar message [ 6875.128956] Lustre: Skipped 1 previous similar message [ 6878.790581] Lustre: server umount lustre-OST0000 complete [ 6885.344682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6885.359651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6885.365806] Lustre: Skipped 1 previous similar message [ 6887.085332] Lustre: server umount lustre-MDT0000 complete [ 6889.081384] LustreError: 406478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365847 with bad export cookie 11408891718083840949 [ 6889.082257] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6889.088829] LustreError: 406478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6889.252218] Lustre: server umount lustre-MDT0001 complete [ 6894.311053] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6895.691417] Key type lgssc unregistered [ 6895.857775] LNet: 410223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6895.864615] LNetError: 410223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6895.875318] LNet: Removed LNI 192.168.203.158@tcp [ 6896.300757] Key type .llcrypt unregistered [ 6896.303122] Key type ._llcrypt unregistered [ 6906.720770] Key type ._llcrypt registered [ 6906.722155] Key type .llcrypt registered [ 6906.767444] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6906.891654] Key type .llcrypt unregistered [ 6906.893321] Key type ._llcrypt unregistered [ 6914.830090] Key type ._llcrypt registered [ 6914.831855] Key type .llcrypt registered [ 6914.879604] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6915.246600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6915.291817] alg: No test for adler32 (adler32-zlib) [ 6916.173206] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6916.285842] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6917.881853] Key type lgssc registered [ 6918.346222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6921.500975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6921.513053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.710958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6924.366606] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6927.596315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6927.727707] LustreError: 412405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6927.733639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.736511] LustreError: 412405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6929.885638] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6932.512534] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6932.834078] LustreError: 412410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.843976] LustreError: 412410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6932.851067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6935.140418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6935.293214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6936.673143] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6945.761724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6945.763101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6945.764241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6945.771247] Lustre: Skipped 1 previous similar message [ 6950.100345] Lustre: server umount lustre-OST0000 complete [ 6954.977415] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6954.980082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.984241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6954.986467] Lustre: Skipped 1 previous similar message [ 6956.000040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6956.000703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.526191] Lustre: server umount lustre-MDT0000 complete [ 6958.764251] LustreError: 412391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365917 with bad export cookie 1341427866200775303 [ 6958.767358] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6958.769056] LustreError: 412391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6958.889840] Lustre: server umount lustre-MDT0001 complete [ 6962.643055] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6963.863471] Key type lgssc unregistered [ 6964.000390] LNet: 415556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6964.004663] LNetError: 415556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6964.011320] LNet: Removed LNI 192.168.203.158@tcp [ 6964.364801] Key type .llcrypt unregistered [ 6964.366499] Key type ._llcrypt unregistered [ 6974.689128] Key type ._llcrypt registered [ 6974.690307] Key type .llcrypt registered [ 6974.740401] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6975.107596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6975.157090] alg: No test for adler32 (adler32-zlib) [ 6976.039116] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 6976.145091] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6977.735149] Key type lgssc registered [ 6978.113956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6980.944571] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6980.953239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.384332] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7001.503372] LustreError: 416924:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.158@tcp: failed processing log, type 4: rc = -110 [ 7028.127624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.740278] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7033.692559] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7036.407955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.617235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7036.647917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7038.436382] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7042.016810] LustreError: 416949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7042.023682] LustreError: 416949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7042.323731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7042.471671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7044.344473] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7047.477820] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7049.968951] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7053.793740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7053.796755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7053.800657] Lustre: Skipped 1 previous similar message [ 7053.806245] Lustre: Skipped 1 previous similar message [ 7058.913324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7058.916585] Lustre: Skipped 1 previous similar message [ 7059.855098] Lustre: server umount lustre-OST0000 complete [ 7063.007908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7063.010794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.016339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7064.035921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7067.370202] Lustre: server umount lustre-MDT0000 complete [ 7068.927156] LustreError: 419745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366027 with bad export cookie 4050348548834223830 [ 7068.931020] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7068.932827] LustreError: 419745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7069.072256] Lustre: server umount lustre-MDT0001 complete [ 7073.350116] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7074.576788] Key type lgssc unregistered [ 7074.725674] LNet: 420494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7074.729827] LNetError: 420494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7074.737564] LNet: Removed LNI 192.168.203.158@tcp [ 7075.110927] Key type .llcrypt unregistered [ 7075.113061] Key type ._llcrypt unregistered [ 7085.634807] Key type ._llcrypt registered [ 7085.636529] Key type .llcrypt registered [ 7085.682869] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7086.080140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7086.136360] alg: No test for adler32 (adler32-zlib) [ 7087.018431] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 7087.133838] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7088.727145] Key type lgssc registered [ 7089.116149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7092.101361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7092.112284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.300610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7094.916676] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7098.087755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7098.224131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7099.725896] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7102.618596] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7104.746858] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7107.160789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7107.288566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7109.394663] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7112.129719] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7118.565661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7123.937886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.940622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7123.942950] Lustre: Skipped 2 previous similar messages [ 7123.945473] Lustre: Skipped 1 previous similar message [ 7123.948495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7127.602260] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 7129.056932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7129.062491] Lustre: Skipped 2 previous similar messages [ 7129.686785] Lustre: server umount lustre-MDT0000 complete [ 7130.492528] LustreError: 421868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.58@tcp arrived at 1788366089 with bad export cookie 3960401341576885121 [ 7130.498235] LustreError: 421868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7131.180303] LustreError: 423726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366090 with bad export cookie 3960401341576884470 [ 7131.184160] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7131.187575] LustreError: 423726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7134.176133] LustreError: 421887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.176577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7134.176880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.177450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7134.182596] LustreError: 421887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7136.223869] LustreError: 421888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7136.231127] LustreError: 421888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7137.296186] Lustre: server umount lustre-OST0000 complete [ 7139.645282] Lustre: server umount lustre-MDT0001 complete [ 7143.499456] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7144.679065] Key type lgssc unregistered [ 7144.811704] LNet: 425586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7144.814525] LNetError: 425586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7144.824342] LNet: Removed LNI 192.168.203.158@tcp [ 7145.185053] Key type .llcrypt unregistered [ 7145.186823] Key type ._llcrypt unregistered [ 7154.916364] Key type ._llcrypt registered [ 7154.917810] Key type .llcrypt registered [ 7154.961827] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7155.423532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7155.431517] alg: No test for adler32 (adler32-zlib) [ 7156.287737] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 7156.385552] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7157.967231] Key type lgssc registered [ 7158.465471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7162.109860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7162.124684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.369619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.978976] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7172.626375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.168774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7177.146555] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7182.360143] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7187.357968] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7192.305910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7192.695790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7197.061514] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7203.354665] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7210.119117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7220.196513] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7220.216853] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7220.256477] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7224.265541] Lustre: server umount lustre-MDT0001 complete [ 7225.395029] LustreError: 428587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7225.434460] LustreError: 428587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7230.949253] LustreError: 428587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7230.965924] LustreError: 428587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7236.064671] LustreError: 426980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7236.067053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7236.067433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.067439] Lustre: Skipped 1 previous similar message [ 7236.068132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7236.068135] Lustre: Skipped 2 previous similar messages [ 7238.269522] Lustre: server umount lustre-OST0000 complete [ 7241.191135] LustreError: 426981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7242.315473] Lustre: server umount lustre-MDT0000 complete [ 7251.830145] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7253.562956] Key type lgssc unregistered [ 7253.769381] LNet: 430680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7253.777937] LNetError: 430680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7253.795116] LNet: Removed LNI 192.168.203.158@tcp [ 7254.292723] Key type .llcrypt unregistered [ 7254.294742] Key type ._llcrypt unregistered [ 7268.569821] Key type ._llcrypt registered [ 7268.571676] Key type .llcrypt registered [ 7268.636452] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 7279.130952] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7279.948127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7279.981503] alg: No test for adler32 (adler32-zlib) [ 7280.996881] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 7281.166426] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7282.767175] Key type lgssc registered [ 7283.548531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7308.215978] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7318.113961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7318.137781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.310277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7319.329303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7319.397352] Lustre: lustre-MDT0000: new disk, initializing [ 7319.456660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7319.470302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7321.927382] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7331.778583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7331.952882] Lustre: lustre-OST0000: new disk, initializing [ 7331.955719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7331.958753] Lustre: Skipped 1 previous similar message [ 7331.962301] Lustre: 435689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7332.007263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.588330] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7335.788931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7335.798460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7335.815236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7341.303701] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7350.418837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7350.502789] Lustre: 434671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7350.543819] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7350.584335] Lustre: lustre-MDT0001: new disk, initializing [ 7350.637172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7350.657222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7350.662378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7352.944364] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7358.836392] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7358.972044] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7361.696447] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7361.846427] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7366.216680] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.58@tcp (stopping) [ 7368.672501] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7368.678859] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.680512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7368.688764] Lustre: Skipped 1 previous similar message [ 7368.701393] Lustre: Skipped 1 previous similar message [ 7371.115415] Lustre: server umount lustre-MDT0001 complete [ 7378.914712] LustreError: 435697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7378.924887] LustreError: 435697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7384.033646] LustreError: 437683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.049656] LustreError: 437683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7387.105681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7387.112977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.124216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7389.153220] LustreError: 434685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.153648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7391.740320] Lustre: server umount lustre-OST0000 complete [ 7400.439890] Lustre: server umount lustre-MDT0000 complete [ 7407.840407] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7409.404301] Key type lgssc unregistered [ 7409.566725] LNet: 439433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7409.571197] LNetError: 439433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7409.583138] LNet: Removed LNI 192.168.203.158@tcp [ 7410.055863] Key type .llcrypt unregistered [ 7410.061303] Key type ._llcrypt unregistered [ 7432.324708] Key type ._llcrypt registered [ 7432.326577] Key type .llcrypt registered [ 7432.387964] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7432.521591] Key type .llcrypt unregistered [ 7432.523877] Key type ._llcrypt unregistered [ 7442.369780] Key type ._llcrypt registered [ 7442.372971] Key type .llcrypt registered [ 7442.441508] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7443.190786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7443.265193] alg: No test for adler32 (adler32-zlib) [ 7444.212768] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 7444.346768] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7445.959157] Key type lgssc registered [ 7446.687614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7462.090115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7462.102469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.230339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7463.256931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7463.300853] Lustre: lustre-MDT0000: new disk, initializing [ 7463.382882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7463.398129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7466.498858] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7474.176972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.263765] Lustre: 442783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7474.328704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7474.331855] Lustre: Skipped 1 previous similar message [ 7474.385197] Lustre: lustre-MDT0001: new disk, initializing [ 7474.456422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7474.496498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7474.510559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7477.979659] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7485.085609] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7488.902710] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7493.103339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.308875] Lustre: lustre-OST0000: new disk, initializing [ 7493.313035] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7493.321354] Lustre: 444751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7493.369398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.106180] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7500.800881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7500.806382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7500.868510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7502.671433] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7510.496295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7510.504427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.513271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7511.014248] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7511.026628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7511.030623] Lustre: Skipped 2 previous similar messages [ 7512.602817] Lustre: server umount lustre-MDT0000 complete [ 7514.783494] LustreError: 443577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366473 with bad export cookie 13362648855378487355 [ 7514.787967] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7514.791112] LustreError: 443577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7515.024245] Lustre: server umount lustre-MDT0001 complete [ 7527.739404] Lustre: server umount lustre-OST0000 complete [ 7533.827271] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:28:11 (1788366491) [ 7538.269992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.573741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7540.896640] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7543.777357] LustreError: 446776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7543.790619] LustreError: 446776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7546.005027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.261675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7548.842739] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7553.859604] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7557.404657] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7574.190157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7574.369829] Lustre: Found index 0 for lustre-OST0000, updating log [ 7574.505588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.059430] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7582.529252] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7584.755815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7591.973593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.981362] Lustre: Skipped 1 previous similar message [ 7591.985631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7597.091300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7597.101426] Lustre: Skipped 3 previous similar messages [ 7603.171030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7603.175445] Lustre: Skipped 1 previous similar message [ 7604.191617] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7604.390909] Lustre: server umount lustre-OST0000 complete [ 7618.078619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7618.089590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7623.137096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7623.137214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.351157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7632.436276] Lustre: server umount lustre-MDT0000 complete [ 7633.378689] LustreError: 446776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7633.391027] LustreError: 446776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7634.401812] LustreError: 446773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.416497] LustreError: 446773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7634.503879] LustreError: 448618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366593 with bad export cookie 13362648855378488748 [ 7634.506165] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.517510] LustreError: 448618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7634.747659] Lustre: server umount lustre-MDT0001 complete [ 7639.786787] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:29:57 (1788366597) [ 7644.063411] LNet: 1 local NIs in recovery (showing 1): 192.168.203.158@tcp [ 8236.192435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8236.471604] LustreError: 453184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8236.537740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8239.394365] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8241.632328] LustreError: 453185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8244.389275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8244.656873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8247.975849] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8252.868480] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8256.159682] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8273.258701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8273.448097] Lustre: Found index 0 for lustre-OST0000, updating log [ 8273.648597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8277.735068] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8283.228246] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8284.882530] Lustre: server umount lustre-OST0000 complete [ 8290.784089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8290.794650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8290.813085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8290.822750] Lustre: Skipped 4 previous similar messages [ 8291.876392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8292.899080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8292.904437] Lustre: Skipped 3 previous similar messages [ 8293.836886] Lustre: server umount lustre-MDT0000 complete [ 8293.926469] LustreError: 455492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8293.938882] LustreError: 455492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8296.578679] LustreError: 453166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367255 with bad export cookie 13362648855378489798 [ 8296.581132] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8296.587326] LustreError: 453166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8296.810055] Lustre: server umount lustre-MDT0001 complete [ 8297.376431] LNet: 1 local NIs in recovery (showing 1): 192.168.203.158@tcp [ 8319.984637] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 8326.616188] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8350.387692] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:41:48 (1788367308) [ 8361.659000] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 8363.372359] Key type lgssc unregistered [ 8363.583316] LNet: 459793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8363.597936] LNetError: 459793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8402.537453] LNet: Removed LNI 192.168.203.158@tcp [ 8403.094925] Key type .llcrypt unregistered [ 8403.097051] Key type ._llcrypt unregistered [ 8413.684542] Key type ._llcrypt registered [ 8413.687533] Key type .llcrypt registered [ 8413.754928] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8414.290269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8414.341218] alg: No test for adler32 (adler32-zlib) [ 8415.258702] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 8415.407661] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8417.023188] Key type lgssc registered [ 8417.721615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8420.509975] LNet: Added LNI 192.168.203.158@tcp100 [8/256/0/180] [ 8421.124045] LNet: Added LNI 192.168.203.158@tcp101 [8/256/0/180] [ 8421.610169] LNet: Added LNI 192.168.203.158@tcp102 [8/256/0/180] [ 8422.167892] LNet: Added LNI 192.168.203.158@tcp103 [8/256/0/180] [ 8422.680183] LNet: Added LNI 192.168.203.158@tcp104 [8/256/0/180] [ 8423.257938] LNet: Added LNI 192.168.203.158@tcp105 [8/256/0/180] [ 8423.867655] LNet: Added LNI 192.168.203.158@tcp106 [8/256/0/180] [ 8424.316299] LNet: Added LNI 192.168.203.158@tcp107 [8/256/0/180] [ 8424.751792] LNet: Added LNI 192.168.203.158@tcp108 [8/256/0/180] [ 8425.284522] LNet: Added LNI 192.168.203.158@tcp109 [8/256/0/180] [ 8425.796411] LNet: Added LNI 192.168.203.158@tcp110 [8/256/0/180] [ 8426.327696] LNet: Added LNI 192.168.203.158@tcp111 [8/256/0/180] [ 8426.898126] LNet: Added LNI 192.168.203.158@tcp112 [8/256/0/180] [ 8427.448739] LNet: Added LNI 192.168.203.158@tcp113 [8/256/0/180] [ 8428.019303] LNet: Added LNI 192.168.203.158@tcp114 [8/256/0/180] [ 8428.609805] LNet: Added LNI 192.168.203.158@tcp115 [8/256/0/180] [ 8429.188761] LNet: Added LNI 192.168.203.158@tcp116 [8/256/0/180] [ 8429.732804] LNet: Added LNI 192.168.203.158@tcp117 [8/256/0/180] [ 8430.270724] LNet: Added LNI 192.168.203.158@tcp118 [8/256/0/180] [ 8430.916092] LNet: Added LNI 192.168.203.158@tcp119 [8/256/0/180] [ 8431.473784] LNet: Added LNI 192.168.203.158@tcp120 [8/256/0/180] [ 8432.156298] LNet: Added LNI 192.168.203.158@tcp121 [8/256/0/180] [ 8432.810503] LNet: Added LNI 192.168.203.158@tcp122 [8/256/0/180] [ 8433.328453] LNet: Added LNI 192.168.203.158@tcp123 [8/256/0/180] [ 8433.933782] LNet: Added LNI 192.168.203.158@tcp124 [8/256/0/180] [ 8434.540285] LNet: Added LNI 192.168.203.158@tcp125 [8/256/0/180] [ 8435.082896] LNet: Added LNI 192.168.203.158@tcp126 [8/256/0/180] [ 8435.624988] LNet: Added LNI 192.168.203.158@tcp127 [8/256/0/180] [ 8436.128648] LNet: Added LNI 192.168.203.158@tcp128 [8/256/0/180] [ 8436.635850] LNet: Added LNI 192.168.203.158@tcp129 [8/256/0/180] [ 8437.111857] LNet: Added LNI 192.168.203.158@tcp130 [8/256/0/180] [ 8437.682673] LNet: Added LNI 192.168.203.158@tcp131 [8/256/0/180] [ 8438.260904] LNet: Added LNI 192.168.203.158@tcp132 [8/256/0/180] [ 8438.732803] LNet: Added LNI 192.168.203.158@tcp133 [8/256/0/180] [ 8439.263041] LNet: Added LNI 192.168.203.158@tcp134 [8/256/0/180] [ 8439.747400] LNet: Added LNI 192.168.203.158@tcp135 [8/256/0/180] [ 8443.235493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8443.249453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.319315] Lustre: 462787:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.158@tcp: can't add NID 192.168.203.158@tcp130: rc = -75 [ 8444.413184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8444.435499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8444.482796] Lustre: lustre-MDT0000: new disk, initializing [ 8444.545926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8444.558184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8446.736854] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8452.313217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8452.384812] Lustre: 462806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8452.446893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8452.450077] Lustre: Skipped 1 previous similar message [ 8452.491516] Lustre: lustre-MDT0001: new disk, initializing [ 8452.510073] Lustre: 463599:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.158@tcp: can't add NID 192.168.203.158@tcp130: rc = -75 [ 8452.515544] Lustre: 463599:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8452.577265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8452.598943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8452.603950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8454.482800] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8458.250428] Lustre: 464133:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.158@tcp: can't add NID 192.168.203.158@tcp130: rc = -75 [ 8458.257452] Lustre: 464133:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8459.510783] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8462.387176] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8465.897206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8466.053125] Lustre: lustre-OST0000: new disk, initializing [ 8466.055586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8466.059935] Lustre: 464772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8466.115485] Lustre: 464777:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.158@tcp: can't add NID 192.168.203.158@tcp130: rc = -75 [ 8466.121059] Lustre: 464777:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 8466.149990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8469.269933] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8474.111052] Lustre: 465435:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.158@tcp: can't add NID 192.168.203.158@tcp130: rc = -75 [ 8474.115555] Lustre: 465435:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8474.152207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8474.157962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8474.226913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8474.667094] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8479.203215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8479.206852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.227100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8479.716961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8479.718535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.737093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8480.893710] Lustre: server umount lustre-OST0000 complete [ 8483.296115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8483.304220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8483.318598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8489.144393] Lustre: server umount lustre-MDT0000 complete [ 8491.197375] LustreError: 462796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367450 with bad export cookie 16116257699422305358 [ 8491.198109] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8491.208295] LustreError: 462796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8491.407819] Lustre: server umount lustre-MDT0001 complete [ 8496.952950] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 8498.418736] Key type lgssc unregistered [ 8498.629290] LNet: 466681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8498.634778] LNetError: 466681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8498.639210] LNet: Removed LNI 192.168.203.158@tcp [ 8499.679139] LNet: Removed LNI 192.168.203.158@tcp100 [ 8499.684325] LNetError: 466681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8499.687692] LNetError: 466681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8499.694073] LNet: Removed LNI 192.168.203.158@tcp101 [ 8499.698151] LNet: Removed LNI 192.168.203.158@tcp102 [ 8499.700260] LNet: Removed LNI 192.168.203.158@tcp103 [ 8499.702470] LNet: Removed LNI 192.168.203.158@tcp104 [ 8499.704638] LNet: Removed LNI 192.168.203.158@tcp105 [ 8499.706569] LNet: Removed LNI 192.168.203.158@tcp106 [ 8499.708765] LNet: Removed LNI 192.168.203.158@tcp107 [ 8499.710843] LNet: Removed LNI 192.168.203.158@tcp108 [ 8499.713233] LNet: Removed LNI 192.168.203.158@tcp109 [ 8499.715338] LNet: Removed LNI 192.168.203.158@tcp110 [ 8499.717286] LNet: Removed LNI 192.168.203.158@tcp111 [ 8499.719413] LNet: Removed LNI 192.168.203.158@tcp112 [ 8499.721514] LNet: Removed LNI 192.168.203.158@tcp113 [ 8499.723393] LNet: Removed LNI 192.168.203.158@tcp114 [ 8499.725483] LNet: Removed LNI 192.168.203.158@tcp115 [ 8499.727336] LNet: Removed LNI 192.168.203.158@tcp116 [ 8499.728970] LNet: Removed LNI 192.168.203.158@tcp117 [ 8499.730763] LNet: Removed LNI 192.168.203.158@tcp118 [ 8499.732853] LNet: Removed LNI 192.168.203.158@tcp119 [ 8499.734781] LNet: Removed LNI 192.168.203.158@tcp120 [ 8499.736592] LNet: Removed LNI 192.168.203.158@tcp121 [ 8499.738502] LNet: Removed LNI 192.168.203.158@tcp122 [ 8499.740232] LNet: Removed LNI 192.168.203.158@tcp123 [ 8499.742282] LNet: Removed LNI 192.168.203.158@tcp124 [ 8499.744250] LNet: Removed LNI 192.168.203.158@tcp125 [ 8499.745897] LNet: Removed LNI 192.168.203.158@tcp126 [ 8499.747896] LNet: Removed LNI 192.168.203.158@tcp127 [ 8499.749432] LNet: Removed LNI 192.168.203.158@tcp128 [ 8499.751747] LNet: Removed LNI 192.168.203.158@tcp129 [ 8499.753790] LNet: Removed LNI 192.168.203.158@tcp130 [ 8499.755935] LNet: Removed LNI 192.168.203.158@tcp131 [ 8499.757864] LNet: Removed LNI 192.168.203.158@tcp132 [ 8499.759902] LNet: Removed LNI 192.168.203.158@tcp133 [ 8499.761857] LNet: Removed LNI 192.168.203.158@tcp134 [ 8499.770251] LNet: Removed LNI 192.168.203.158@tcp135 [ 8500.211628] Key type .llcrypt unregistered [ 8500.213805] Key type ._llcrypt unregistered [ 8512.355278] Key type ._llcrypt registered [ 8512.356968] Key type .llcrypt registered [ 8512.410686] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 8521.290409] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8521.898852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8521.909232] alg: No test for adler32 (adler32-zlib) [ 8522.844847] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 8522.977035] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8524.599216] Key type lgssc registered [ 8525.190092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8543.809197] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:45:01 (1788367501) [ 8553.027076] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 8554.457835] Key type lgssc unregistered [ 8554.637687] LNet: 470739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8554.642343] LNetError: 470739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8554.659369] LNet: Removed LNI 192.168.203.158@tcp [ 8555.110360] Key type .llcrypt unregistered [ 8555.112300] Key type ._llcrypt unregistered [ 8565.104919] Key type ._llcrypt registered [ 8565.107707] Key type .llcrypt registered [ 8565.177847] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8565.593444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8565.649709] alg: No test for adler32 (adler32-zlib) [ 8566.567865] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 8566.731267] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8568.335155] Key type lgssc registered [ 8568.960250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8573.897030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8573.918456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8575.048200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8575.070686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8575.120900] Lustre: lustre-MDT0000: new disk, initializing [ 8575.169281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.179372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8577.327644] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8583.284567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.349297] Lustre: 471929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8583.381109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8583.384240] Lustre: Skipped 1 previous similar message [ 8583.428212] Lustre: lustre-MDT0001: new disk, initializing [ 8583.489713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8583.514525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8583.521165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8585.693706] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8591.116751] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8594.340668] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8597.651043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8597.794960] Lustre: lustre-OST0000: new disk, initializing [ 8597.797534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8597.801623] Lustre: 473898:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8597.844321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.442061] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8603.650225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8603.654680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8603.706666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8604.927297] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8606.559852] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing lnet_if_list [ 8608.661809] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8608.675893] LNet: Added LNI 192.168.203.158@tcp10 [8/256/0/180] [ 8611.413720] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8611.430576] LNet: Added LNI 192.168.203.158@tcp11 [8/256/0/180] [ 8613.351059] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8613.365553] LNet: Added LNI 192.168.203.158@tcp12 [8/256/0/180] [ 8615.345835] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8615.363097] LNet: Added LNI 192.168.203.158@tcp13 [8/256/0/180] [ 8617.366285] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8617.382590] LNet: Added LNI 192.168.203.158@tcp14 [8/256/0/180] [ 8619.402300] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8619.420072] LNet: Added LNI 192.168.203.158@tcp15 [8/256/0/180] [ 8621.704145] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8621.720649] LNet: Added LNI 192.168.203.158@tcp16 [8/256/0/180] [ 8623.877466] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8623.892040] LNet: Added LNI 192.168.203.158@tcp17 [8/256/0/180] [ 8625.819526] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8625.834285] LNet: Added LNI 192.168.203.158@tcp18 [8/256/0/180] [ 8627.794733] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8627.808935] LNet: Added LNI 192.168.203.158@tcp19 [8/256/0/180] [ 8629.837445] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8629.858199] LNet: Added LNI 192.168.203.158@tcp20 [8/256/0/180] [ 8632.004642] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8632.020438] LNet: Added LNI 192.168.203.158@tcp21 [8/256/0/180] [ 8634.169814] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8634.185949] LNet: Added LNI 192.168.203.158@tcp22 [8/256/0/180] [ 8636.682340] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8636.698238] LNet: Added LNI 192.168.203.158@tcp23 [8/256/0/180] [ 8638.990996] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8639.005698] LNet: Added LNI 192.168.203.158@tcp24 [8/256/0/180] [ 8641.400031] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8641.414382] LNet: Added LNI 192.168.203.158@tcp25 [8/256/0/180] [ 8644.022278] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8644.039601] LNet: Added LNI 192.168.203.158@tcp26 [8/256/0/180] [ 8646.906583] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8646.926381] LNet: Added LNI 192.168.203.158@tcp27 [8/256/0/180] [ 8649.784788] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8649.810647] LNet: Added LNI 192.168.203.158@tcp28 [8/256/0/180] [ 8652.622041] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8652.637825] LNet: Added LNI 192.168.203.158@tcp29 [8/256/0/180] [ 8655.401764] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8655.422945] LNet: Added LNI 192.168.203.158@tcp30 [8/256/0/180] [ 8658.427849] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8658.448775] LNet: Added LNI 192.168.203.158@tcp31 [8/256/0/180] [ 8661.391524] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8661.407730] LNet: Added LNI 192.168.203.158@tcp32 [8/256/0/180] [ 8664.829855] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8664.845652] LNet: Added LNI 192.168.203.158@tcp33 [8/256/0/180] [ 8668.384430] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8668.404203] LNet: Added LNI 192.168.203.158@tcp34 [8/256/0/180] [ 8671.936709] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8671.957198] LNet: Added LNI 192.168.203.158@tcp35 [8/256/0/180] [ 8675.158746] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8675.175480] LNet: Added LNI 192.168.203.158@tcp36 [8/256/0/180] [ 8678.493374] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8678.511518] LNet: Added LNI 192.168.203.158@tcp37 [8/256/0/180] [ 8681.936901] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8681.960976] LNet: Added LNI 192.168.203.158@tcp38 [8/256/0/180] [ 8685.632090] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8685.647865] LNet: Added LNI 192.168.203.158@tcp39 [8/256/0/180] [ 8689.348310] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8689.365573] LNet: Added LNI 192.168.203.158@tcp40 [8/256/0/180] [ 8692.914047] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8692.930341] LNet: Added LNI 192.168.203.158@tcp41 [8/256/0/180] [ 8696.357737] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8696.378811] LNet: Added LNI 192.168.203.158@tcp42 [8/256/0/180] [ 8700.104451] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8700.123084] LNet: Added LNI 192.168.203.158@tcp43 [8/256/0/180] [ 8704.067908] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8704.084593] LNet: Added LNI 192.168.203.158@tcp44 [8/256/0/180] [ 8707.854166] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8707.874668] LNet: Added LNI 192.168.203.158@tcp45 [8/256/0/180] [ 8711.725941] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8711.746624] LNet: Added LNI 192.168.203.158@tcp46 [8/256/0/180] [ 8715.446072] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8715.461616] LNet: Added LNI 192.168.203.158@tcp47 [8/256/0/180] [ 8719.225644] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8719.244447] LNet: Added LNI 192.168.203.158@tcp48 [8/256/0/180] [ 8723.047486] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8723.062302] LNet: Added LNI 192.168.203.158@tcp49 [8/256/0/180] [ 8727.098525] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8727.117853] LNet: Added LNI 192.168.203.158@tcp50 [8/256/0/180] [ 8736.737893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8736.739659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8736.744390] Lustre: Skipped 2 previous similar messages [ 8736.751123] Lustre: Skipped 2 previous similar messages [ 8741.856232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8741.858680] Lustre: Skipped 2 previous similar messages [ 8742.519536] Lustre: server umount lustre-MDT0000 complete [ 8744.136141] LustreError: 471921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367702 with bad export cookie 7284865408385005936 [ 8744.140441] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8744.141352] LustreError: 471921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8744.285491] Lustre: server umount lustre-MDT0001 complete [ 8753.006214] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8756.670978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.335131] Lustre: 471204:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367705/real 1788367705] req@00000000ecf726b6 x1875239265994880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8762.345157] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.407129] Lustre: 471202:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367708/real 1788367708] req@000000002a9ec026 x1875239265995008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8767.455119] Lustre: 471201:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367710/real 1788367710] req@00000000033400d4 x1875239265995136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8768.479415] LustreError: 471200:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005f960838 x1875239265995904/t0(0) o250->MGC192.168.203.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8768.622813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8769.503137] Lustre: 471203:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367712/real 1788367712] req@000000001a6723aa x1875239265995392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8770.477197] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8774.373179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.586763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8775.583155] Lustre: 471204:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788367718/real 1788367718] req@00000000d0d46fbd x1875239265995520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788367734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8776.380519] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8779.649862] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8779.749564] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8782.210132] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8794.592086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8794.598846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.606646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8795.104354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8797.797486] Lustre: server umount lustre-OST0000 complete [ 8800.227435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8800.230031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8800.231400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8800.231408] Lustre: Skipped 2 previous similar messages [ 8800.244889] Lustre: Skipped 2 previous similar messages [ 8805.344183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8805.346830] Lustre: Skipped 2 previous similar messages [ 8805.726916] Lustre: server umount lustre-MDT0000 complete [ 8807.203077] LustreError: 471922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367766 with bad export cookie 7284865408385008183 [ 8807.206787] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8807.207288] LustreError: 471922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8807.359208] Lustre: server umount lustre-MDT0001 complete [ 8812.057056] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 8813.395426] Key type lgssc unregistered [ 8813.581833] LNet: 500868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8813.588441] LNetError: 500868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8813.591687] LNet: Removed LNI 192.168.203.158@tcp [ 8813.593376] LNet: Removed LNI 192.168.203.158@tcp10 [ 8813.595428] LNet: Removed LNI 192.168.203.158@tcp11 [ 8813.597441] LNet: Removed LNI 192.168.203.158@tcp12 [ 8813.599847] LNet: Removed LNI 192.168.203.158@tcp13 [ 8813.602251] LNet: Removed LNI 192.168.203.158@tcp14 [ 8813.604868] LNet: Removed LNI 192.168.203.158@tcp15 [ 8813.606404] LNet: Removed LNI 192.168.203.158@tcp16 [ 8813.607741] LNet: Removed LNI 192.168.203.158@tcp17 [ 8813.609720] LNet: Removed LNI 192.168.203.158@tcp18 [ 8813.611239] LNet: Removed LNI 192.168.203.158@tcp19 [ 8813.613197] LNet: Removed LNI 192.168.203.158@tcp20 [ 8813.615063] LNet: Removed LNI 192.168.203.158@tcp21 [ 8813.617241] LNet: Removed LNI 192.168.203.158@tcp22 [ 8813.619630] LNet: Removed LNI 192.168.203.158@tcp23 [ 8813.622573] LNet: Removed LNI 192.168.203.158@tcp24 [ 8813.626136] LNet: Removed LNI 192.168.203.158@tcp25 [ 8813.628411] LNet: Removed LNI 192.168.203.158@tcp26 [ 8813.630920] LNet: Removed LNI 192.168.203.158@tcp27 [ 8813.632919] LNet: Removed LNI 192.168.203.158@tcp28 [ 8813.634571] LNet: Removed LNI 192.168.203.158@tcp29 [ 8813.637199] LNet: Removed LNI 192.168.203.158@tcp30 [ 8813.639249] LNet: Removed LNI 192.168.203.158@tcp31 [ 8813.641161] LNet: Removed LNI 192.168.203.158@tcp32 [ 8813.643388] LNet: Removed LNI 192.168.203.158@tcp33 [ 8813.645680] LNet: Removed LNI 192.168.203.158@tcp34 [ 8813.647462] LNet: Removed LNI 192.168.203.158@tcp35 [ 8813.649195] LNet: Removed LNI 192.168.203.158@tcp36 [ 8813.653702] LNet: Removed LNI 192.168.203.158@tcp37 [ 8813.656281] LNet: Removed LNI 192.168.203.158@tcp38 [ 8813.658565] LNet: Removed LNI 192.168.203.158@tcp39 [ 8813.660891] LNet: Removed LNI 192.168.203.158@tcp40 [ 8813.662838] LNet: Removed LNI 192.168.203.158@tcp41 [ 8813.664980] LNet: Removed LNI 192.168.203.158@tcp42 [ 8813.667102] LNet: Removed LNI 192.168.203.158@tcp43 [ 8813.669255] LNet: Removed LNI 192.168.203.158@tcp44 [ 8813.671495] LNet: Removed LNI 192.168.203.158@tcp45 [ 8813.673318] LNet: Removed LNI 192.168.203.158@tcp46 [ 8813.674922] LNet: Removed LNI 192.168.203.158@tcp47 [ 8813.676822] LNet: Removed LNI 192.168.203.158@tcp48 [ 8813.678973] LNet: Removed LNI 192.168.203.158@tcp49 [ 8813.685728] LNet: Removed LNI 192.168.203.158@tcp50 [ 8814.026194] Key type .llcrypt unregistered [ 8814.028061] Key type ._llcrypt unregistered [ 8822.143318] Key type ._llcrypt registered [ 8822.144891] Key type .llcrypt registered [ 8822.208391] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 8829.446033] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8829.861438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8829.924689] alg: No test for adler32 (adler32-zlib) [ 8830.816808] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 8830.946430] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8832.551385] Key type lgssc registered [ 8833.090251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8847.524668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8847.534434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.632990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8848.647210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8848.690308] Lustre: lustre-MDT0000: new disk, initializing [ 8848.721673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8848.731894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8850.492359] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8855.746994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.801922] Lustre: 504055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8855.834272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8855.840120] Lustre: Skipped 1 previous similar message [ 8855.884357] Lustre: lustre-MDT0001: new disk, initializing [ 8855.928439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8855.945711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8855.954268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8857.783280] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8862.436881] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8864.748638] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8867.880301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8867.962069] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8868.001605] Lustre: lustre-OST0000: new disk, initializing [ 8868.004335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8868.010813] Lustre: 506071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8868.065665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8870.588793] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8875.400729] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8876.580885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8876.585548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8876.655027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8881.636381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8881.638145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8881.641241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8881.649758] Lustre: Skipped 1 previous similar message [ 8881.656639] Lustre: Skipped 1 previous similar message [ 8884.961492] Lustre: server umount lustre-OST0000 complete [ 8886.753023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8886.754240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.761523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8886.766448] Lustre: Skipped 1 previous similar message [ 8891.875681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8891.881741] Lustre: Skipped 2 previous similar messages [ 8892.798085] Lustre: server umount lustre-MDT0000 complete [ 8894.504611] LustreError: 504048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367853 with bad export cookie 11142187708462532205 [ 8894.508487] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8894.512347] LustreError: 504048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8894.669697] Lustre: server umount lustre-MDT0001 complete [ 8899.406232] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 8900.680786] Key type lgssc unregistered [ 8900.830563] LNet: 507976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8900.837047] LNetError: 507976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8900.845509] LNet: Removed LNI 192.168.203.158@tcp [ 8901.240772] Key type .llcrypt unregistered [ 8901.242614] Key type ._llcrypt unregistered [ 8910.609762] Key type ._llcrypt registered [ 8910.611485] Key type .llcrypt registered [ 8910.680163] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8911.255324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8911.269366] alg: No test for adler32 (adler32-zlib) [ 8912.158605] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 8912.274993] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8913.879145] Key type lgssc registered [ 8914.372590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8917.782206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8917.793974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.031465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8920.736736] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8924.429037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8924.605496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8926.242031] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8929.355837] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8931.663104] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8934.539623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8934.702584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8937.265639] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8940.484062] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8949.949255] loop0: detected capacity change from 0 to 204800000 [ 8950.020093] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8950.024828] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8950.029626] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8952.619691] loop0: detected capacity change from 0 to 204800000 [ 8952.682680] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8952.690219] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8952.695670] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8954.805016] loop0: detected capacity change from 0 to 204800000 [ 8957.113578] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8957.180552] Lustre: fs15246-OST0000: new disk, initializing [ 8957.183163] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8957.187553] Lustre: 512818:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8957.214944] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8959.691119] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8964.408709] loop1: detected capacity change from 0 to 204800000 [ 8966.859572] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.910747] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8966.938528] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8966.978706] Lustre: fs15246-MDT0000: new disk, initializing [ 8967.049219] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.062758] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8967.112556] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8968.160751] Lustre: 513937:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b87755bd x1875239628735488/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788367943 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8968.813225] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8977.251423] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8977.255189] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8977.263623] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8977.964138] Lustre: server umount fs15246-MDT0000 complete [ 8981.478040] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8981.486628] LustreError: 509220:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8981.494027] LustreError: 509220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8983.876827] Lustre: server umount fs15246-OST0000 complete [ 8990.176939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8990.181384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8990.186404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8991.713665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8991.714857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8994.747647] Lustre: server umount lustre-OST0000 complete [ 8996.833136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8996.840685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8996.843908] Lustre: Skipped 1 previous similar message [ 9001.440084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9001.445082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.451962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9002.721645] Lustre: server umount lustre-MDT0000 complete [ 9004.393161] LustreError: 509204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367963 with bad export cookie 2353329724221630920 [ 9004.397687] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9004.398683] LustreError: 509204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9004.537733] Lustre: server umount lustre-MDT0001 complete [ 9009.064151] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 9010.325626] Key type lgssc unregistered [ 9010.486472] LNet: 517029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9010.492533] LNetError: 517029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9010.504213] LNet: Removed LNI 192.168.203.158@tcp [ 9010.901190] Key type .llcrypt unregistered [ 9010.902726] Key type ._llcrypt unregistered [ 9019.732853] Key type ._llcrypt registered [ 9019.734344] Key type .llcrypt registered [ 9019.795272] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9020.244987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9020.263584] alg: No test for adler32 (adler32-zlib) [ 9021.150586] Lustre: Lustre: Build Version: 2.17.56_1_g896c54c [ 9021.274208] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 9022.887380] Key type lgssc registered [ 9023.411132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9039.509531] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 9043.321627] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9057.196417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9057.203733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9058.289867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9058.301915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9058.338437] Lustre: lustre-MDT0000: new disk, initializing [ 9058.373309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9058.381762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9059.943579] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9064.866200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.925776] Lustre: 521467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9064.963309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9064.966134] Lustre: Skipped 1 previous similar message [ 9065.008171] Lustre: lustre-MDT0001: new disk, initializing [ 9065.059430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9065.076796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9065.080350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9066.825777] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9071.367851] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9073.626449] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9076.541424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9076.667408] Lustre: lustre-OST0000: new disk, initializing [ 9076.669988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9076.674559] Lustre: 523432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9076.727399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9079.253338] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9082.883366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9082.889818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9082.934804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9083.567376] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9085.213188] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9085.294791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9087.020726] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9087.115753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9087.969092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9087.970156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9087.972464] LustreError: Skipped 1 previous similar message [ 9087.979587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9087.985977] Lustre: Skipped 1 previous similar message [ 9093.092576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9093.095855] Lustre: Skipped 1 previous similar message [ 9098.210300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9098.212901] Lustre: Skipped 1 previous similar message [ 9102.303163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9102.371489] Lustre: server umount lustre-OST0000 complete [ 9105.888929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9105.892761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9105.898885] Lustre: Skipped 1 previous similar message [ 9105.901633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9108.450403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9110.247162] Lustre: server umount lustre-MDT0000 complete [ 9111.893422] LustreError: 521459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788368070 with bad export cookie 11618571388466518743 [ 9111.896771] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9111.900949] LustreError: 521459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9112.045141] Lustre: server umount lustre-MDT0001 complete [ 9121.905106] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 9126.046350] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9137.972130] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8811 sec ========== 12:54:56 (1788368096) [ 9138.646126] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:54:56 (1788368096) === [ 9139.915934] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:54:58 (1788368098) === [ 9154.673367] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 9155.845433] Key type lgssc unregistered [ 9155.994461] LNet: 530063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9155.997582] LNetError: 530063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9156.004287] LNet: Removed LNI 192.168.203.158@tcp [ 9156.323605] Key type .llcrypt unregistered [ 9156.325347] Key type ._llcrypt unregistered