[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 504146327 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003347] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008516] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011149] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.014038] SELinux: Initializing. [ 0.014984] *** VALIDATE selinux *** [ 0.023242] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028150] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031116] *** VALIDATE tmpfs *** [ 0.033253] *** VALIDATE proc *** [ 0.034234] *** VALIDATE cgroup *** [ 0.035007] *** VALIDATE cgroup2 *** [ 0.036269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042613] debug: unmapping init [mem 0xffffffffbca59000-0xffffffffbca60fff] [ 0.044203] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045728] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047021] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052019] ... event mask: 000000070000000f [ 0.053295] rcu: Hierarchical SRCU implementation. [ 0.055344] smp: Bringing up secondary CPUs ... [ 0.056623] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060473] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133996] node 0 deferred pages initialised in 67ms [ 0.136097] devtmpfs: initialized [ 0.138225] x86/mm: Memory block size: 128MB [ 0.141996] gcov: version magic: 0x41383552 [ 0.144262] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151304] pinctrl core: initialized pinctrl subsystem [ 0.153172] [ 0.153756] ************************************************************* [ 0.156011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159013] ** ** [ 0.162014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164010] ** ** [ 0.167013] ** This means that this kernel is built to expose internal ** [ 0.169009] ** IOMMU data structures, which may compromise security on ** [ 0.172018] ** your system. ** [ 0.174010] ** ** [ 0.177013] ** If you see this message and you are not debugging the ** [ 0.179011] ** kernel, report this immediately to your vendor! ** [ 0.182013] ** ** [ 0.184012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187012] ************************************************************* [ 0.189681] NET: Registered protocol family 16 [ 0.192428] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200448] cpuidle: using governor menu [ 0.201407] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203403] PCI: Using configuration type 1 for base access [ 0.206127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215039] cryptd: max_cpu_qlen set to 1000 [ 0.217191] ACPI: Added _OSI(Module Device) [ 0.219022] ACPI: Added _OSI(Processor Device) [ 0.221022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227718] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235549] ACPI: Interpreter enabled [ 0.237073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238014] ACPI: Using IOAPIC for interrupt routing [ 0.241097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245424] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268422] acpiphp: Slot [2] registered [ 0.270307] acpiphp: Slot [5] registered [ 0.273203] acpiphp: Slot [6] registered [ 0.274144] acpiphp: Slot [7] registered [ 0.276161] acpiphp: Slot [8] registered [ 0.278188] acpiphp: Slot [9] registered [ 0.280165] acpiphp: Slot [10] registered [ 0.282192] acpiphp: Slot [3] registered [ 0.284156] acpiphp: Slot [4] registered [ 0.286215] acpiphp: Slot [11] registered [ 0.288135] acpiphp: Slot [12] registered [ 0.289123] acpiphp: Slot [13] registered [ 0.291121] acpiphp: Slot [14] registered [ 0.293211] acpiphp: Slot [15] registered [ 0.295114] acpiphp: Slot [16] registered [ 0.297096] acpiphp: Slot [17] registered [ 0.298120] acpiphp: Slot [18] registered [ 0.300114] acpiphp: Slot [19] registered [ 0.302116] acpiphp: Slot [20] registered [ 0.303160] acpiphp: Slot [21] registered [ 0.305124] acpiphp: Slot [22] registered [ 0.306139] acpiphp: Slot [23] registered [ 0.308080] acpiphp: Slot [24] registered [ 0.309086] acpiphp: Slot [25] registered [ 0.311085] acpiphp: Slot [26] registered [ 0.312123] acpiphp: Slot [27] registered [ 0.314110] acpiphp: Slot [28] registered [ 0.316103] acpiphp: Slot [29] registered [ 0.317125] acpiphp: Slot [30] registered [ 0.319120] acpiphp: Slot [31] registered [ 0.320042] PCI host bridge to bus 0000:00 [ 0.322019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.331023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.333018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.339184] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342415] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352846] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.359674] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.367015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370557] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373782] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377738] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401777] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408230] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.449938] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.477022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488301] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.499020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.506020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.516024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.524457] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.529933] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.535019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.556019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.564087] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.573023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.578019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.595019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.610629] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.618022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.625032] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.642025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.653000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.654444] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.656428] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.658838] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.660189] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.665153] iommu: Default domain type: Passthrough [ 0.667548] SCSI subsystem initialized [ 0.670229] ACPI: bus type USB registered [ 0.671168] usbcore: registered new interface driver usbfs [ 0.673099] usbcore: registered new interface driver hub [ 0.675086] usbcore: registered new device driver usb [ 0.676214] pps_core: LinuxPPS API ver. 1 registered [ 0.678018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.681092] PTP clock support registered [ 0.683161] EDAC MC: Ver: 3.0.0 [ 0.685563] PCI: Using ACPI for IRQ routing [ 0.687565] NetLabel: Initializing [ 0.689017] NetLabel: domain hash size = 128 [ 0.691014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.693082] NetLabel: unlabeled traffic allowed by default [ 0.697158] vgaarb: loaded [ 0.699282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.700019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.704214] clocksource: Switched to clocksource kvm-clock [ 0.814534] VFS: Disk quotas dquot_6.6.0 [ 0.816409] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.819244] *** VALIDATE ramfs *** [ 0.821101] *** VALIDATE hugetlbfs *** [ 0.822450] pnp: PnP ACPI init [ 0.824935] pnp: PnP ACPI: found 6 devices [ 0.843666] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.848749] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.851768] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.854628] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.856827] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.859289] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.863269] NET: Registered protocol family 2 [ 0.866265] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.873968] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.877780] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.884265] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.888350] TCP: Hash tables configured (established 65536 bind 65536) [ 0.891728] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.894628] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.898498] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.902561] NET: Registered protocol family 1 [ 0.907283] RPC: Registered named UNIX socket transport module. [ 0.910124] RPC: Registered udp transport module. [ 0.912622] RPC: Registered tcp transport module. [ 0.914993] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.917245] NET: Registered protocol family 44 [ 0.918986] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.921577] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.923932] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.927028] PCI: CLS 0 bytes, default 64 [ 0.928939] Unpacking initramfs... [ 4.721321] debug: unmapping init [mem 0xffff95cf7cc54000-0xffff95cf7ffbffff] [ 4.730288] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.734877] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.753760] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.214934] Initialise system trusted keyrings [ 6.220909] Key type blacklist registered [ 6.228947] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.257726] zbud: loaded [ 6.271514] *** VALIDATE nfs *** [ 6.277287] *** VALIDATE nfs4 *** [ 6.283838] pstore: using deflate compression [ 6.341161] Platform Keyring initialized [ 6.744917] NET: Registered protocol family 38 [ 6.748221] Key type asymmetric registered [ 6.753366] Asymmetric key parser 'x509' registered [ 6.758270] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.767232] io scheduler mq-deadline registered [ 6.771980] io scheduler kyber registered [ 6.775879] io scheduler bfq registered [ 6.778792] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.785667] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.793573] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.798657] ACPI: Power Button [PWRF] [ 6.808515] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.818740] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.855373] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.871754] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.909165] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.961393] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.094543] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.174333] Non-volatile memory driver v1.3 [ 7.186463] Linux agpgart interface v0.103 [ 7.335536] virtio_blk virtio1: [vda] 150088 512-byte logical blocks (76.8 MB/73.3 MiB) [ 7.344776] vda: detected capacity change from 0 to 76845056 [ 7.374493] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.379273] vdb: detected capacity change from 0 to 1073741824 [ 7.403790] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.406324] vdc: detected capacity change from 0 to 2621440000 [ 7.426948] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.431445] vdd: detected capacity change from 0 to 2621440000 [ 7.451694] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.455743] vde: detected capacity change from 0 to 4294967296 [ 7.471856] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.475585] vdf: detected capacity change from 0 to 4294967296 [ 7.482599] libphy: Fixed MDIO Bus: probed [ 7.512967] usbcore: registered new interface driver usbserial_generic [ 7.519371] usbserial: USB Serial support registered for generic [ 7.523313] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.531640] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.534466] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.539660] mousedev: PS/2 mouse device common for all mice [ 7.545773] rtc_cmos 00:05: RTC can wake from S4 [ 7.553448] rtc_cmos 00:05: registered as rtc0 [ 7.557339] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.561202] intel_pstate: CPU model not supported [ 7.569783] hid: raw HID events driver (C) Jiri Kosina [ 7.571342] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.574596] usbcore: registered new interface driver usbhid [ 7.574603] usbhid: USB HID core driver [ 7.574759] drop_monitor: Initializing network drop monitor service [ 7.574892] Initializing XFRM netlink socket [ 7.576521] NET: Registered protocol family 10 [ 7.600202] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.608847] Segment Routing with IPv6 [ 7.612140] NET: Registered protocol family 17 [ 7.614821] mpls_gso: MPLS GSO support [ 7.621793] RAS: Correctable Errors collector initialized. [ 7.626376] AVX version of gcm_enc/dec engaged. [ 7.628821] AES CTR mode by8 optimization enabled [ 7.630283] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.807412] sched_clock: Marking stable (7807249329, 0)->(8765885320, -958635991) [ 7.818514] registered taskstats version 1 [ 7.826949] Loading compiled-in X.509 certificates [ 7.831271] zswap: loaded using pool lzo/zbud [ 7.892404] Key type big_key registered [ 7.932437] Key type encrypted registered [ 7.933573] ima: No TPM chip found, activating TPM-bypass! [ 7.935323] ima: Allocated hash algorithm: sha1 [ 7.949098] ima: No architecture policies found [ 7.950546] evm: Initialising EVM extended attributes: [ 7.951903] evm: security.selinux [ 7.959065] evm: security.ima [ 7.961659] evm: security.capability [ 7.964181] evm: HMAC attrs: 0x1 [ 7.977409] rtc_cmos 00:05: setting system clock to 2026-09-04 08:18:54 UTC (1788509934) [ 7.988404] debug: unmapping init [mem 0xffffffffbda03000-0xffffffffbdbfffff] [ 7.993217] debug: unmapping init [mem 0xffffffffbc782000-0xffffffffbca58fff] [ 8.002337] Write protecting the kernel read-only data: 28672k [ 8.043928] debug: unmapping init [mem 0xffffffffbae03000-0xffffffffbaffffff] [ 8.053527] debug: unmapping init [mem 0xffffffffbb714000-0xffffffffbb7fffff] [ 8.181444] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.213643] systemd[1]: Detected virtualization kvm. [ 8.217992] systemd[1]: Detected architecture x86-64. [ 8.224852] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.322553] systemd[1]: No hostname configured. [ 8.324407] systemd[1]: Set hostname to . [ 8.329550] random: systemd: uninitialized urandom read (16 bytes read) [ 8.335362] systemd[1]: Initializing machine ID from random generator. [ 8.677814] random: systemd: uninitialized urandom read (16 bytes read) [ 8.681491] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 8.691649] random: systemd: uninitialized urandom read (16 bytes read) [ 8.697129] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 8.707841] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.438660] device-mapper: uevent: version 1.0.3 [ 10.446216] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.307111] virtio_net virtio0 ens2: renamed from eth0 [ 12.351328] random: fast init done [ 12.508741] scsi host0: ata_piix [ 12.549570] scsi host1: ata_piix [ 12.551661] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.560500] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.510465] random: crng init done [ 17.515432] random: 7 urandom warning(s) missed due to ratelimiting [ 17.958944] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.197702] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.419639] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.316819] SELinux: Disabled at runtime. [ 24.447437] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.463400] systemd[1]: Detected virtualization kvm. [ 24.467891] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.174643] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.180200] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.192448] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.199474] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.205986] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.218579] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.234432] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ 26.281657] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 27.692272] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.811920] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.847703] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.463529] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.506796] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.360254] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.027420] hrtimer: interrupt took 7275728 ns [ 35.207708] NFS: Registering the id_resolver key type [ 35.213293] Key type id_resolver registered [ 35.217807] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg351-server login: [ 91.255175] libcfs: loading out-of-tree module taints kernel. [ 91.310569] Key type ._llcrypt registered [ 91.314605] Key type .llcrypt registered [ 91.417930] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 110.227902] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 111.687232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.706256] alg: No test for adler32 (adler32-zlib) [ 113.216263] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 114.204622] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 115.968152] Key type lgssc registered [ 117.597145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.536437] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 178.048297] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 192.147934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.179914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.511357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.575449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.689579] Lustre: lustre-MDT0000: new disk, initializing [ 193.761786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.775817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.677330] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.687135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.804213] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.833573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.839295] Lustre: Skipped 1 previous similar message [ 210.930554] Lustre: lustre-MDT0001: new disk, initializing [ 210.990363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.030810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 211.040782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.342686] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.216488] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 229.875123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.172987] Lustre: lustre-OST0000: new disk, initializing [ 230.180132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 230.191725] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.277378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.425930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 233.442255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.597465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.810952] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.151410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.256517] Lustre: lustre-OST0001: new disk, initializing [ 252.260399] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 252.267973] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.350977] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.732481] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.141472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.171432] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 259.343315] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.387417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 275.731297] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 282.731569] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing check_logdir /tmp/testlogs/ [ 288.648451] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing yml_node [ 292.639055] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 295.112889] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 298.222562] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 300.228877] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 04:23:44 EDT 2026 [ 318.154409] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 319.673223] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.922757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 338.931138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.949429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.433184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.440606] Lustre: Skipped 1 previous similar message [ 341.872236] Lustre: server umount lustre-MDT0000 complete [ 345.850589] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788510272 with bad export cookie 2430398559801707120 [ 345.851587] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.870838] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 349.673760] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.683423] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.687794] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.687804] Lustre: Skipped 4 previous similar messages [ 349.700379] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 349.761791] Lustre: Skipped 2 previous similar messages [ 350.689055] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.692135] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.728322] Lustre: Skipped 2 previous similar messages [ 352.654943] Lustre: server umount lustre-MDT0001 complete [ 357.002251] Lustre: server umount lustre-OST0000 complete [ 361.188799] Lustre: server umount lustre-OST0001 complete [ 368.685851] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 380.409203] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 418.670150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.900657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.934188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 419.071398] Lustre: lustre-MDT0000: new disk, initializing [ 419.174558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.206644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 423.562736] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 435.334208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.448840] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 435.508557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 435.512767] Lustre: Skipped 1 previous similar message [ 435.573951] Lustre: lustre-MDT0001: new disk, initializing [ 435.669692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 435.714437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 435.729410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 440.583907] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 450.634970] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 457.028233] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 465.950731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.210836] Lustre: lustre-OST0000: new disk, initializing [ 466.216227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 466.223561] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 466.322726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.288709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 468.298225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 468.437063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 473.126165] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 483.216212] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 487.766635] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 487.982600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.934256] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 493.127285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 499.169949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 499.174098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.187841] LustreError: Skipped 1 previous similar message [ 499.189396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.228750] Lustre: Skipped 1 previous similar message [ 501.282230] Lustre: server umount lustre-OST0000 complete [ 507.363117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 507.378442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.396098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.405280] Lustre: Skipped 1 previous similar message [ 511.722122] Lustre: server umount lustre-MDT0000 complete [ 514.529422] LustreError: 15078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.549106] LustreError: 15078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 515.861702] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788510442 with bad export cookie 2430398559801708653 [ 515.872810] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.627454] Lustre: server umount lustre-MDT0001 complete [ 521.296434] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:27:25 (1788510445) [ 523.684352] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 525.649879] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:27:29 (1788510449) [ 532.230422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.729775] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.810776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.548437] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 538.082556] LustreError: 19450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.056687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.976482] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 556.717148] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 562.477437] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 570.170040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 578.889118] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 586.483254] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 597.483112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 597.492226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.511964] Lustre: Skipped 1 previous similar message [ 597.521955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 597.530768] Lustre: Skipped 2 previous similar messages [ 600.422988] Lustre: server umount lustre-OST0000 complete [ 606.690212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 606.704482] LustreError: Skipped 1 previous similar message [ 610.873269] Lustre: server umount lustre-MDT0000 complete [ 613.861226] LustreError: 19450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.886978] LustreError: 19450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 615.142290] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788510541 with bad export cookie 2430398559801709528 [ 615.147474] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.168720] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 615.647459] Lustre: server umount lustre-MDT0001 complete [ 628.366874] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 632.645439] Key type lgssc unregistered [ 632.991235] LNet: 22981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.010375] LNetError: 22981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.036939] LNet: Removed LNI 192.168.203.151@tcp [ 633.952165] Key type .llcrypt unregistered [ 633.954586] Key type ._llcrypt unregistered [ 657.794898] Key type ._llcrypt registered [ 657.806080] Key type .llcrypt registered [ 657.967730] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 659.373645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.527558] alg: No test for adler32 (adler32-zlib) [ 660.579923] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 660.881424] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 662.633393] Key type lgssc registered [ 663.681802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 671.641489] loop: module loaded [ 671.713566] loop0: detected capacity change from 0 to 204800000 [ 671.934643] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 671.954884] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 672.230284] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.021131] loop0: detected capacity change from 0 to 204800000 [ 678.158561] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.183145] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.189990] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.743891] loop0: detected capacity change from 0 to 204800000 [ 690.134654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 690.180851] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.584397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 691.709475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 691.854665] Lustre: lustre-MDT0000: new disk, initializing [ 691.995094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 692.020123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 696.947383] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 706.702607] loop1: detected capacity change from 0 to 204800000 [ 712.632486] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 712.962355] Lustre: lustre-OST07c6: new disk, initializing [ 712.966438] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 712.974045] Lustre: Skipped 1 previous similar message [ 712.979930] Lustre: 26167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 713.076583] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 719.642525] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 721.577961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 721.595107] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 721.634446] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 724.646845] Lustre: 25154:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 724.672979] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 724.679630] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 724.692577] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 724.701584] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 724.708651] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 752.874808] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 752.893510] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 752.907954] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.926585] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 752.940029] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 752.946603] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.677127] Lustre: 26834:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 758.683024] Lustre: 26834:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 758.687196] Lustre: 26834:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 758.691289] Lustre: 26834:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.696993] Lustre: 26834:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.703971] Lustre: 26834:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.722887] Lustre: 26834:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 758.738355] Lustre: 26834:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.747516] Lustre: 26834:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 758.755232] Lustre: 26834:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.764746] Lustre: 26834:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.774415] Lustre: 26834:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.942492] Lustre: 25154:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 761.952290] Lustre: 25154:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 761.956502] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 761.960681] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.964602] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 761.968952] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.972823] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 761.976401] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.980251] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 761.984548] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.989188] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 761.993915] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.715409] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 766.731162] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 766.747707] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 766.755169] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.770039] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.783779] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.799668] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 766.804848] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.812233] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 766.829698] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.840792] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 766.846768] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.735128] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 777.746745] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 777.754396] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.761334] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.769682] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.783341] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.796344] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 777.805714] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.815668] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.827693] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 777.835049] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.844904] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 828.902920] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 828.911880] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.931832] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 834.032507] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 839.159481] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 840.160563] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 840.350915] Lustre: server umount lustre-OST07c6 complete [ 850.506737] Lustre: server umount lustre-MDT0000 complete [ 865.768923] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 868.792225] Key type lgssc unregistered [ 869.104683] LNet: 28746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.109783] LNetError: 28746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.183310] LNet: Removed LNI 192.168.203.151@tcp [ 871.114888] Key type .llcrypt unregistered [ 871.118857] Key type ._llcrypt unregistered [ 890.700366] Key type ._llcrypt registered [ 890.703227] Key type .llcrypt registered [ 890.845598] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 892.127211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 892.215298] alg: No test for adler32 (adler32-zlib) [ 893.789587] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 894.234619] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 896.017328] Key type lgssc registered [ 897.526892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 907.299445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 907.355457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.862375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.150122] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 919.015224] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 919.041039] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 923.072607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.455773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 927.707154] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 935.430996] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 941.250689] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 947.443425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.758795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.444339] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 953.900168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 960.807992] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 978.810731] Lustre: server umount lustre-OST0000 complete [ 984.560018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.564322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 984.589087] Lustre: Skipped 1 previous similar message [ 984.608093] Lustre: Skipped 1 previous similar message [ 989.669494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.678884] Lustre: Skipped 1 previous similar message [ 992.839158] Lustre: server umount lustre-MDT0000 complete [ 994.785428] LustreError: 30015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.786472] Lustre: 29322:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788510905/real 1788510905] req@00000000d5e834f4 x1875389521623168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788510921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 994.812236] LustreError: 30015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 994.874892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.849446] LustreError: 29996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788510923 with bad export cookie 10748419189557459252 [ 996.854492] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.867601] LustreError: 29996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 997.224185] Lustre: server umount lustre-MDT0001 complete [ 1008.251883] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1011.144824] Key type lgssc unregistered [ 1011.396497] LNet: 33748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.405688] LNetError: 33748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.420321] LNet: Removed LNI 192.168.203.151@tcp [ 1012.366583] Key type .llcrypt unregistered [ 1012.369164] Key type ._llcrypt unregistered [ 1033.550417] Key type ._llcrypt registered [ 1033.552299] Key type .llcrypt registered [ 1033.672446] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1034.661415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1034.799836] alg: No test for adler32 (adler32-zlib) [ 1035.870313] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 1036.154226] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1037.808215] Key type lgssc registered [ 1038.905496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1046.829691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1046.845694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.332722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.375289] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1057.764518] LustreError: 35016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.786644] LustreError: 35016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1060.489154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.767390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1065.365278] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1071.732949] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1076.596355] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1083.771429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1084.178650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.832909] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1093.436981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1098.782446] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1112.048473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1112.061150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.088183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1114.083231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1114.087772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1114.092151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.104788] Lustre: Skipped 1 previous similar message [ 1115.856772] Lustre: server umount lustre-OST0000 complete [ 1122.273312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1122.285291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.312111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1124.324355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1124.324411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.863575] Lustre: server umount lustre-MDT0000 complete [ 1129.442800] LustreError: 37311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.472069] LustreError: 37311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1130.798720] LustreError: 34997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788511057 with bad export cookie 6742523923753116402 [ 1130.813975] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.820097] LustreError: 34997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1131.155148] Lustre: server umount lustre-MDT0001 complete [ 1142.081390] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1145.065040] Key type lgssc unregistered [ 1145.355744] LNet: 38556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.369741] LNetError: 38556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.394069] LNet: Removed LNI 192.168.203.151@tcp [ 1146.335801] Key type .llcrypt unregistered [ 1146.342078] Key type ._llcrypt unregistered [ 1169.984574] Key type ._llcrypt registered [ 1169.987236] Key type .llcrypt registered [ 1170.117984] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1170.885679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1170.911986] alg: No test for adler32 (adler32-zlib) [ 1171.918437] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 1172.172327] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1173.864349] Key type lgssc registered [ 1175.099332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1182.739865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1182.754391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.181197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.309446] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1194.467797] LustreError: 39968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1194.492211] LustreError: 39968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1196.772264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.028796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1201.137498] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1208.558354] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1214.769341] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1221.773217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1222.220445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.911366] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1229.432093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1236.737786] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1247.204565] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.206260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.219527] Lustre: Skipped 1 previous similar message [ 1249.703162] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 1249.721798] Lustre: Skipped 2 previous similar messages [ 1252.328950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.337724] Lustre: Skipped 2 previous similar messages [ 1253.491121] Lustre: server umount lustre-MDT0000 complete [ 1254.759459] LustreError: 41777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.450071] LustreError: 39967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.487241] LustreError: 39967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1258.006943] LustreError: 40702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788511184 with bad export cookie 5269076315189458134 [ 1258.029048] LustreError: 40702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1258.034347] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.881119] LustreError: 41777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.927474] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [ 1262.577404] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.606321] Lustre: Skipped 1 previous similar message [ 1264.547733] Lustre: server umount lustre-MDT0001 complete [ 1279.772840] Lustre: server umount lustre-OST0000 complete [ 1294.359679] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1296.930324] Key type lgssc unregistered [ 1297.210128] LNet: 43825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.221326] LNetError: 43825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.245262] LNet: Removed LNI 192.168.203.151@tcp [ 1298.131980] Key type .llcrypt unregistered [ 1298.134273] Key type ._llcrypt unregistered [ 1319.724512] Key type ._llcrypt registered [ 1319.728727] Key type .llcrypt registered [ 1319.826839] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1320.774146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.789851] alg: No test for adler32 (adler32-zlib) [ 1321.938871] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 1322.176711] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1323.840467] Key type lgssc registered [ 1324.876360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.150697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.165345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.617178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.035320] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1343.970544] LustreError: 45237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.001643] LustreError: 45237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1345.521597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.907358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.061109] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1356.402770] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1363.007672] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1371.365945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1371.777157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.940895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1379.870059] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1387.328377] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1399.777455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1399.785645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.805930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1401.835327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1401.837359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1401.837429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.840699] Lustre: Skipped 2 previous similar messages [ 1403.345697] Lustre: server umount lustre-OST0000 complete [ 1422.305522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.319870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.768272] Lustre: server umount lustre-MDT0000 complete [ 1428.450415] LustreError: 45967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.470626] LustreError: 45967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1430.881840] LustreError: 45218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788511357 with bad export cookie 4042476584447389775 [ 1430.889964] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1430.899792] LustreError: 45218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1445.344286] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1445.654985] Lustre: server umount lustre-MDT0001 complete [ 1455.719626] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1457.838863] Key type lgssc unregistered [ 1458.041732] LNet: 48946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.053590] LNetError: 48946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.062559] LNet: Removed LNI 192.168.203.151@tcp [ 1458.682161] Key type .llcrypt unregistered [ 1458.683954] Key type ._llcrypt unregistered [ 1479.927866] Key type ._llcrypt registered [ 1479.946159] Key type .llcrypt registered [ 1480.075947] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1480.971041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1481.012107] alg: No test for adler32 (adler32-zlib) [ 1482.114683] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 1482.337159] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1483.984151] Key type lgssc registered [ 1484.870802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.337672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1492.370273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.817414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.779487] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1504.262600] LustreError: 50357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.279442] LustreError: 50357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1508.078235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.463785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1512.648598] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1519.512680] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1525.307997] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1532.629365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1533.081577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.244474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1539.245862] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1546.760584] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1553.343480] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1557.217226] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1569.387347] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-04 4h44m53s [ 1570.824806] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1574.888749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1574.908876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.920975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1575.971525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1575.978308] Lustre: Skipped 3 previous similar messages [ 1578.898521] Lustre: server umount lustre-MDT0000 complete [ 1580.397295] LustreError: 50352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.054090] LustreError: 50358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.068538] LustreError: 50358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1582.354694] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1586.551394] LustreError: 53130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1588.244771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.374201] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1588.603377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.664639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1592.745515] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1594.866176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1595.818712] Lustre: DEBUG MARKER: Wait for df (36677) ... [ 1597.592720] Lustre: DEBUG MARKER: done [ 1604.065712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1604.075193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.084839] Lustre: Skipped 2 previous similar messages [ 1604.089869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1604.096717] Lustre: Skipped 1 previous similar message [ 1605.092966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.684659] Lustre: server umount lustre-OST0000 complete [ 1615.331913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1615.337092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.347937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.347944] Lustre: Skipped 2 previous similar messages [ 1615.390187] Lustre: Skipped 1 previous similar message [ 1622.498454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1622.505433] Lustre: Skipped 4 previous similar messages [ 1625.056092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1625.324717] Lustre: server umount lustre-MDT0000 complete [ 1625.571112] LustreError: 50358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.593051] LustreError: 50358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1628.872145] LustreError: 52987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788511555 with bad export cookie 8840092686909171836 [ 1628.873161] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1628.880158] LustreError: 52987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1629.180892] Lustre: server umount lustre-MDT0001 complete [ 1639.495352] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1641.976707] Key type lgssc unregistered [ 1642.271680] LNet: 55283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.285944] LNetError: 55283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.308804] LNet: Removed LNI 192.168.203.151@tcp [ 1643.157138] Key type .llcrypt unregistered [ 1643.159166] Key type ._llcrypt unregistered [ 1687.107485] Key type ._llcrypt registered [ 1687.109109] Key type .llcrypt registered [ 1687.275182] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1687.806698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1687.826287] alg: No test for adler32 (adler32-zlib) [ 1689.094151] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 1689.546484] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1691.360247] Key type lgssc registered [ 1692.787741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1701.543923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1701.555800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.830843] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1702.870522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1703.031785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.412592] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1716.430852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.593971] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1716.651872] Lustre: 57487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1717.039979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1723.654885] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1735.994918] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1744.770935] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1752.454084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1752.717553] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1753.043758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.887677] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1763.008284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1766.170991] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1773.098602] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-04 4h48m17s [ 1774.628704] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1781.705660] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1784.039571] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1784.126047] LustreError: 57495:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1804.194545] Lustre: lustre-MDT0000: Client 5219a24f-b20a-4ceb-9e29-a33dd26d3c16 (at 192.168.203.51@tcp) reconnecting [ 1824.153770] LustreError: 57495:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1827.107251] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1830.666126] Lustre: DEBUG MARKER: done [ 1839.586603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.598996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.619215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.099375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.114849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.133553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.135986] Lustre: Skipped 1 previous similar message [ 1844.797775] Lustre: server umount lustre-OST0000 complete [ 1850.337715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.342717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.349826] Lustre: Skipped 1 previous similar message [ 1855.195247] Lustre: server umount lustre-MDT0000 complete [ 1858.495061] LustreError: 57478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788511785 with bad export cookie 5318196196026657387 [ 1858.501789] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1858.522476] LustreError: 57478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1858.708804] Lustre: server umount lustre-MDT0001 complete [ 1869.190491] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1871.937750] Key type lgssc unregistered [ 1872.357720] LNet: 61820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.366617] LNetError: 61820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.382140] LNet: Removed LNI 192.168.203.151@tcp [ 1873.311861] Key type .llcrypt unregistered [ 1873.317608] Key type ._llcrypt unregistered [ 1921.511840] Key type ._llcrypt registered [ 1921.514301] Key type .llcrypt registered [ 1921.612553] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1922.603858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1922.697333] alg: No test for adler32 (adler32-zlib) [ 1923.881430] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 1924.133477] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1925.849331] Key type lgssc registered [ 1926.874612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1929.966501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1929.995922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.252302] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1931.282858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1931.430053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.413532] Lustre: Failing over lustre-MDT0000 [ 1932.654394] Lustre: server umount lustre-MDT0000 complete [ 1938.964943] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:51:03 (1788511863) [ 1945.435815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.859707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.249970] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1958.303921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.373714] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1958.397590] Lustre: 64617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1958.665898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1962.919566] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1970.778664] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1976.684723] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1983.297434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1983.532828] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1983.773289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.235342] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1991.486827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1999.397190] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2004.726880] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2011.079493] Lustre: server umount lustre-MDT0000 complete [ 2012.137218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.156662] Lustre: Skipped 1 previous similar message [ 2014.732974] LustreError: 65350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788511941 with bad export cookie 12843959994266435152 [ 2014.733339] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2014.745326] LustreError: 65350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2015.107284] Lustre: server umount lustre-MDT0001 complete [ 2018.545904] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2022.704156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2032.612774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.377730] Lustre: 63539:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788511943/real 1788511943] req@00000000529bb80c x1875390601665152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788511959 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2033.403883] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.420473] Lustre: Skipped 1 previous similar message [ 2034.656156] Lustre: 63541:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788511945/real 1788511945] req@000000009f48d6dc x1875390601665280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788511961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2037.792546] Lustre: 63541:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788511948/real 1788511948] req@00000000519e65d7 x1875390601665408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788511964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2039.624058] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb23ee99087944733 [ 2039.642520] Lustre: MGC192.168.203.151@tcp: Connection restored to 0@lo (at 0@lo) [ 2040.057479] LustreError: 66464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.080152] LustreError: 66464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2040.168839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.301057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2041.121110] Lustre: 63538:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788511951/real 1788511951] req@00000000ac34202a x1875390601665664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788511967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2043.245987] LustreError: 66465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.263343] LustreError: 66465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2045.443878] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2047.328186] Lustre: 63538:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788511957/real 1788511957] req@000000006241797f x1875390601665792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788511973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2048.489117] LustreError: 68386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.525122] LustreError: 68386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2051.573709] LustreError: 68386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.578777] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2051.611784] LustreError: 68386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2054.508192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.952256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2059.540518] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2060.302111] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2066.676048] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2072.707104] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2080.738949] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.743219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2080.743828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2080.765798] Lustre: Skipped 1 previous similar message [ 2085.356542] Lustre: server umount lustre-MDT0000 complete [ 2085.859652] LustreError: 66464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.891250] LustreError: 66464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2089.412784] LustreError: 65350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788512015 with bad export cookie 12843959994266437427 [ 2089.422550] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2089.432720] LustreError: 65350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2089.807866] Lustre: server umount lustre-MDT0001 complete [ 2096.788911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2107.360161] Lustre: 63539:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788512017/real 1788512017] req@00000000a2595546 x1875390601691648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788512033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2107.406598] Lustre: 63539:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2107.417892] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.435261] Lustre: Skipped 1 previous similar message [ 2108.311275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.647796] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb23ee99087944c96 [ 2113.670268] Lustre: MGC192.168.203.151@tcp: Connection restored to 0@lo (at 0@lo) [ 2114.073064] LustreError: 66464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.135498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.193492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2118.521844] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2128.247672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.396974] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2134.030080] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2143.474575] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2150.783327] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2159.075459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2159.092369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.128975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.131920] Lustre: Skipped 2 previous similar messages [ 2161.638876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2161.656880] Lustre: Skipped 2 previous similar messages [ 2164.597058] Lustre: server umount lustre-MDT0000 complete [ 2166.754960] LustreError: 66464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.783532] LustreError: 66464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2167.675370] LustreError: 64608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788512094 with bad export cookie 12843959994266438806 [ 2167.681336] LustreError: 64608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2167.685313] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2168.079094] Lustre: server umount lustre-MDT0001 complete [ 2174.099826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2177.376207] Lustre: DEBUG MARKER: files compared the same [ 2189.442187] Lustre: server umount lustre-OST0000 complete [ 2205.426831] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2208.318381] Key type lgssc unregistered [ 2208.769479] LNet: 74345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.778858] LNetError: 74345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.811243] LNet: Removed LNI 192.168.203.151@tcp [ 2209.749392] Key type .llcrypt unregistered [ 2209.756196] Key type ._llcrypt unregistered [ 2241.314736] Key type ._llcrypt registered [ 2241.319044] Key type .llcrypt registered [ 2241.448609] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2242.643712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.683794] alg: No test for adler32 (adler32-zlib) [ 2243.712890] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 2243.967752] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2245.632154] Key type lgssc registered [ 2246.679630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2254.683609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2254.714328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.212150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.087483] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug +malloc all [ 2265.583440] LustreError: 75760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.610981] LustreError: 75760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2269.400858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.748598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2274.037676] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug +malloc all [ 2281.420760] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2286.947180] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2292.969285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.287883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.323258] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug +malloc all [ 2301.506424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2306.743679] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2315.753044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2315.760716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.770630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.770190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2316.771023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.772724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2321.116893] Lustre: server umount lustre-OST0000 complete [ 2325.987828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2326.002576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.027437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2326.029926] Lustre: Skipped 1 previous similar message [ 2330.871975] Lustre: server umount lustre-MDT0000 complete [ 2332.130123] LustreError: 75757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.156381] LustreError: 75757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2335.594631] LustreError: 75740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788512262 with bad export cookie 10033032602911325447 [ 2335.597263] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.608742] LustreError: 75740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2336.124226] Lustre: server umount lustre-MDT0001 complete [ 2348.729826] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2351.872845] Key type lgssc unregistered [ 2352.114780] LNet: 79290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.130506] LNetError: 79290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.170294] LNet: Removed LNI 192.168.203.151@tcp [ 2353.006148] Key type .llcrypt unregistered [ 2353.009622] Key type ._llcrypt unregistered [ 2376.260248] Key type ._llcrypt registered [ 2376.262710] Key type .llcrypt registered [ 2376.360894] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2377.699587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2377.744508] alg: No test for adler32 (adler32-zlib) [ 2378.786869] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 2379.111621] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2380.864187] Key type lgssc registered [ 2381.874191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2388.751617] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2388.769176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2404.256378] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2409.376563] LustreError: 80677:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.151@tcp: failed processing log, type 4: rc = -110 [ 2436.128598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.663872] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2452.218709] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2459.899018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.066861] Lustre: *** cfs_fail_loc=706, val=0*** [ 2460.505739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.584118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2464.793996] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2465.775584] LustreError: 80703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.799454] LustreError: 80703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2470.883918] LustreError: 80702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.915674] LustreError: 80702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2473.954796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.491430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2478.797744] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2485.220428] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2490.268384] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2494.957596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2494.959175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.970480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2494.974656] Lustre: Skipped 1 previous similar message [ 2494.997400] Lustre: Skipped 1 previous similar message [ 2498.664081] Lustre: server umount lustre-OST0000 complete [ 2505.186479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2505.186679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.202384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2505.214049] Lustre: Skipped 1 previous similar message [ 2508.643258] Lustre: server umount lustre-MDT0000 complete [ 2510.310591] LustreError: 81768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.324833] LustreError: 81768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2511.701236] LustreError: 80683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788512438 with bad export cookie 14631735838250060510 [ 2511.717785] LustreError: 80683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2511.720612] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2511.999687] Lustre: server umount lustre-MDT0001 complete [ 2522.101890] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2524.644443] Key type lgssc unregistered [ 2524.950240] LNet: 84293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2524.959569] LNetError: 84293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.982395] LNet: Removed LNI 192.168.203.151@tcp [ 2525.729917] Key type .llcrypt unregistered [ 2525.736308] Key type ._llcrypt unregistered [ 2547.280580] Key type ._llcrypt registered [ 2547.283926] Key type .llcrypt registered [ 2547.410213] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2548.377660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2548.396130] alg: No test for adler32 (adler32-zlib) [ 2549.617883] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 2549.941441] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2551.657270] Key type lgssc registered [ 2552.890697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2560.428989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2560.452667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.917189] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2574.543494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.015897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.195058] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2580.450348] LustreError: 86406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.475887] LustreError: 86406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2584.552773] LustreError: 86404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.589995] LustreError: 86404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2589.571790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.665735] LustreError: 86406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.692767] LustreError: 86406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2589.848316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.307684] LustreError: 86405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.337201] LustreError: 86405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2596.203375] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2600.422405] LustreError: 86410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.461414] LustreError: 86410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2605.453892] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2605.485750] Lustre: MGS: Client 518becfe-618c-46b0-9c37-b3721a85c2b3 (at 0@lo) reconnecting [ 2605.513553] LustreError: 86420:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2605.808901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.876773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2610.387882] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2626.542265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.551205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2626.553109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2626.570061] Lustre: Skipped 1 previous similar message [ 2626.592210] Lustre: Skipped 1 previous similar message [ 2630.441837] Lustre: server umount lustre-OST0000 complete [ 2636.770223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2636.777682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.778509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.792496] Lustre: Skipped 1 previous similar message [ 2636.804766] Lustre: Skipped 1 previous similar message [ 2640.276072] Lustre: server umount lustre-MDT0000 complete [ 2641.889783] LustreError: 86406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.916562] LustreError: 86406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2644.283899] Lustre: server umount lustre-MDT0001 complete [ 2647.773416] Lustre: server umount MGS complete [ 2659.208705] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:03:03 (1788512583) [ 2690.894734] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 2701.477843] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2749.766843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.017546] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2766.689966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.001652] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2767.070199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2767.198715] Lustre: lustre-MDT0001: new disk, initializing [ 2767.324576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2767.353318] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2767.366068] Lustre: 93592:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2767.375086] LustreError: 93592:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2772.038823] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2782.700717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.994135] Lustre: lustre-OST0000: new disk, initializing [ 2782.997228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2783.000949] Lustre: 94358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2783.052667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.759500] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2789.925569] Lustre: 94653:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000095d90d11 x1875391258074112/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788512732 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2799.487570] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2807.942799] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2807.956306] Lustre: MGS: Client 6065a976-523a-4236-be7a-af8e85786a6f (at 0@lo) reconnecting [ 2807.983815] LustreError: 93578:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2808.076228] Lustre: 92859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2808.127559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2808.190259] Lustre: lustre-MDT0000: new disk, initializing [ 2808.334245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.371357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2808.412661] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2809.522194] Lustre: 95428:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f289097d x1875391258080896/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788512752 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2814.021640] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2814.694786] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2817.519395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2817.539133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2818.619414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2818.630374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2818.810210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2841.068860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.080334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2841.097793] Lustre: Skipped 1 previous similar message [ 2846.179197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2846.183248] Lustre: Skipped 2 previous similar messages [ 2846.885985] Lustre: server umount lustre-OST0000 complete [ 2851.300715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.304969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.327951] Lustre: Skipped 1 previous similar message [ 2851.328290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2851.371134] Lustre: Skipped 1 previous similar message [ 2857.180174] Lustre: server umount lustre-MDT0000 complete [ 2868.033974] Lustre: server umount lustre-MDT0001 complete [ 2871.834654] Lustre: server umount MGS complete [ 2883.667528] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:06:47 (1788512807) [ 2902.416178] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2905.139909] Key type lgssc unregistered [ 2905.443770] LNet: 97924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2905.454525] LNetError: 97924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2906.471461] LNet: Removed LNI 192.168.203.151@tcp [ 2907.219372] Key type .llcrypt unregistered [ 2907.221214] Key type ._llcrypt unregistered [ 2925.171662] Key type ._llcrypt registered [ 2925.174371] Key type .llcrypt registered [ 2925.368325] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2926.764462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2926.953889] alg: No test for adler32 (adler32-zlib) [ 2928.052541] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 2928.375300] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2930.097239] Key type lgssc registered [ 2931.831983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2941.000607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2941.031183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.789222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.141899] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2953.194910] LustreError: 99145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.220862] LustreError: 99145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2957.189537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.796666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2962.320991] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2969.145335] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2974.997724] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2982.179650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.638908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.765416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2989.365912] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2996.846392] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3006.793211] Lustre: server umount lustre-OST0000 complete [ 3013.603713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.625058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.723609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.735328] Lustre: Skipped 2 previous similar messages [ 3020.946276] Lustre: server umount lustre-MDT0000 complete [ 3023.842514] LustreError: 100580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.868201] LustreError: 100580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3024.644300] LustreError: 99125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788512951 with bad export cookie 3407164842607687766 [ 3024.649219] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3024.658858] LustreError: 99125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3035.241343] Lustre: server umount lustre-MDT0001 complete [ 3046.630620] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3049.079445] Key type lgssc unregistered [ 3049.425533] LNet: 102678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3049.443535] LNetError: 102678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3050.470069] LNet: Removed LNI 192.168.203.151@tcp [ 3051.224774] Key type .llcrypt unregistered [ 3051.227618] Key type ._llcrypt unregistered [ 3068.562942] Key type ._llcrypt registered [ 3068.569577] Key type .llcrypt registered [ 3068.825769] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3070.083095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3070.092791] alg: No test for adler32 (adler32-zlib) [ 3071.193515] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 3071.451910] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3073.239946] Key type lgssc registered [ 3074.698251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3083.811435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3083.816172] LustreError: 103775:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3083.826125] Lustre: Skipped 1 previous similar message [ 3083.826144] LustreError: 103776:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3083.849173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.851335] LustreError: 103775:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3083.884734] LustreError: 103775:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3083.890905] LustreError: 103775:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3085.605369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.922900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.347314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3098.093635] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3104.113652] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing lsmod [ 3107.117545] LustreError: 104980:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3107.132350] LustreError: 104981:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3107.141881] LustreError: 104980:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3107.156656] LustreError: 104980:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3107.164369] LustreError: 104980:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3107.197810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.618400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.733571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3110.828277] LustreError: 103833:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 3113.953762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3113.966568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.991662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3116.002303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.005918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3116.013123] Lustre: Skipped 1 previous similar message [ 3116.028963] Lustre: Skipped 2 previous similar messages [ 3118.062432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.072525] Lustre: Skipped 1 previous similar message [ 3123.176827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.184199] Lustre: Skipped 2 previous similar messages [ 3125.216139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3125.421293] Lustre: server umount lustre-MDT0000 complete [ 3128.309837] LustreError: 103814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.334366] LustreError: 103814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3130.176849] LustreError: 103789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788513056 with bad export cookie 18034062404170689097 [ 3130.182708] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3130.187118] LustreError: 103789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3130.355422] LustreError: 104244:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000205a6141 x1875391804365184/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3130.639641] Lustre: server umount lustre-MDT0001 complete [ 3145.388968] Lustre: server umount lustre-OST0000 complete [ 3154.197362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.709201] LustreError: 106046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.817263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.040559] LustreError: 106047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.254639] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3165.161102] LustreError: 106046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.274812] LustreError: 106047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.316965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.949561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3177.507590] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3186.496272] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3193.156824] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3200.679276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3201.244168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.162070] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3215.601907] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3224.844684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3235.298080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3235.300636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.321168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.349175] Lustre: Skipped 1 previous similar message [ 3239.793396] Lustre: server umount lustre-OST0000 complete [ 3245.537862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.541708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.570854] Lustre: Skipped 1 previous similar message [ 3245.580929] Lustre: Skipped 1 previous similar message [ 3250.539410] Lustre: server umount lustre-MDT0000 complete [ 3250.657835] LustreError: 106041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.684495] LustreError: 106041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3254.188022] LustreError: 106029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788513180 with bad export cookie 18034062404170690014 [ 3254.198784] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3254.200228] LustreError: 106029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3254.657850] Lustre: server umount lustre-MDT0001 complete [ 3267.063060] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3269.677812] Key type lgssc unregistered [ 3269.965149] LNet: 109601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3269.978522] LNetError: 109601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3271.015327] LNet: Removed LNI 192.168.203.151@tcp [ 3271.861751] Key type .llcrypt unregistered [ 3271.865121] Key type ._llcrypt unregistered [ 3296.602541] Key type ._llcrypt registered [ 3296.604304] Key type .llcrypt registered [ 3296.700592] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3297.937615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3298.066977] alg: No test for adler32 (adler32-zlib) [ 3299.106599] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 3299.385272] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3301.049887] Key type lgssc registered [ 3302.277934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3309.489359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3309.525575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.910322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.237938] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3321.317138] LustreError: 111015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.332908] LustreError: 111015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3323.871356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.331620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3328.711261] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3336.937229] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3343.329638] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3350.469092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3350.891312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.021398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3356.867929] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3364.909062] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3374.166796] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3377.922634] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3384.801675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3384.805734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.816878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3384.826700] Lustre: Skipped 1 previous similar message [ 3389.276284] Lustre: server umount lustre-OST0000 complete [ 3395.044057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.050373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3395.057907] Lustre: Skipped 1 previous similar message [ 3395.062204] Lustre: Skipped 1 previous similar message [ 3400.162227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.172358] Lustre: Skipped 1 previous similar message [ 3405.287249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.291839] Lustre: Skipped 2 previous similar messages [ 3407.329859] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3407.500566] Lustre: server umount lustre-MDT0000 complete [ 3410.409959] LustreError: 111010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.425377] LustreError: 111010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3411.412574] LustreError: 113549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788513337 with bad export cookie 8365140526965791764 [ 3411.420078] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3411.422925] LustreError: 113549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3411.735875] Lustre: server umount lustre-MDT0001 complete [ 3426.931106] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3430.928573] Key type lgssc unregistered [ 3431.263875] LNet: 114658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3431.296446] LNetError: 114658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3431.333335] LNet: Removed LNI 192.168.203.151@tcp [ 3432.555522] Key type .llcrypt unregistered [ 3432.557642] Key type ._llcrypt unregistered [ 3452.589658] Key type ._llcrypt registered [ 3452.594424] Key type .llcrypt registered [ 3452.728428] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3454.000932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3454.200514] alg: No test for adler32 (adler32-zlib) [ 3455.234876] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 3455.548323] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3457.312186] Key type lgssc registered [ 3458.505281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3466.383106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3466.424806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.905850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.052100] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3477.548178] LustreError: 115876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3477.585166] LustreError: 115876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3481.798899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.347865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3487.536820] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3497.224191] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3503.369974] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3510.893412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3511.265507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.351775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3518.385448] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3527.384706] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3542.504783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.516244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.546820] Lustre: Skipped 1 previous similar message [ 3542.578281] Lustre: Skipped 1 previous similar message [ 3547.618454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3547.637888] Lustre: Skipped 1 previous similar message [ 3552.772707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3552.796735] Lustre: Skipped 1 previous similar message [ 3556.320138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3556.527320] Lustre: server umount lustre-OST0000 complete [ 3562.981399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.995150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3562.995763] Lustre: Skipped 1 previous similar message [ 3563.027534] Lustre: Skipped 1 previous similar message [ 3568.101095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3576.288382] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3576.570861] Lustre: server umount lustre-MDT0000 complete [ 3578.342161] LustreError: 115875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.355439] LustreError: 115875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3580.881024] LustreError: 115855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788513507 with bad export cookie 18094450744850510598 [ 3580.911860] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3581.451348] Lustre: server umount lustre-MDT0001 complete [ 3594.792519] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3599.212210] Key type lgssc unregistered [ 3599.771863] LNet: 119427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3599.788317] LNetError: 119427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3599.855531] LNet: Removed LNI 192.168.203.151@tcp [ 3601.094847] Key type .llcrypt unregistered [ 3601.099765] Key type ._llcrypt unregistered [ 3629.652419] Key type ._llcrypt registered [ 3629.657930] Key type .llcrypt registered [ 3629.782583] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3630.947985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3630.977507] alg: No test for adler32 (adler32-zlib) [ 3632.103211] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 3632.310773] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3633.984157] Key type lgssc registered [ 3635.020949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3643.413739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3643.425949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3644.949366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3650.000698] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3655.169463] LustreError: 120842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3655.192110] LustreError: 120842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3660.280926] LustreError: 120841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3660.387793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.848395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3666.350367] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3675.698472] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3682.384767] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3691.293715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3691.903259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.394103] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3709.910724] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3715.504912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3724.256623] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3729.560696] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3729.567367] Lustre: Skipped 2 previous similar messages [ 3732.190744] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3732.285455] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3735.605853] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3735.612360] Lustre: Skipped 2 previous similar messages [ 3739.065772] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3739.068354] Lustre: Skipped 1 previous similar message [ 3745.451701] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.51@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3745.463262] Lustre: Skipped 2 previous similar messages [ 3748.941937] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.51@tcp 192.168.0.[2,10]@tcp [ 3757.858177] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.51/24@tcp in log lustre-MDT0000 [ 3757.869221] Lustre: Skipped 2 previous similar messages [ 3766.373668] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.51/24@tcp [ 3766.388433] Lustre: Skipped 1 previous similar message [ 3772.898282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3772.904481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.930869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3777.001821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3777.019153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.036817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3782.117242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3782.134227] Lustre: Skipped 2 previous similar messages [ 3787.232513] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3787.237806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3787.266335] Lustre: Skipped 1 previous similar message [ 3787.554434] Lustre: server umount lustre-OST0000 complete [ 3792.355073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.357079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3792.381749] Lustre: Skipped 1 previous similar message [ 3802.593776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.607119] Lustre: Skipped 4 previous similar messages [ 3806.180392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3806.473528] Lustre: server umount lustre-MDT0000 complete [ 3807.713410] LustreError: 123278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.734855] LustreError: 123278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3810.209506] LustreError: 120821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788513736 with bad export cookie 9097285803733888137 [ 3810.220895] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3810.222492] LustreError: 120821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3810.597577] Lustre: server umount lustre-MDT0001 complete [ 3822.338442] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3825.185982] Key type lgssc unregistered [ 3825.556734] LNet: 125810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3825.563527] LNetError: 125810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3825.602758] LNet: Removed LNI 192.168.203.151@tcp [ 3826.692374] Key type .llcrypt unregistered [ 3826.704363] Key type ._llcrypt unregistered [ 3854.756132] Key type ._llcrypt registered [ 3854.760216] Key type .llcrypt registered [ 3854.838707] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3855.945684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3856.066660] alg: No test for adler32 (adler32-zlib) [ 3857.097938] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 3857.411483] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3859.096234] Key type lgssc registered [ 3860.355148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3868.743892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3868.767498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3870.272268] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3870.307629] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.51/24@tcp [ 3870.317925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.289763] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3880.417682] LustreError: 127367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.440141] LustreError: 127367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3881.948678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.300381] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3882.327466] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.51/24@tcp [ 3882.396329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3886.325292] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3893.462541] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3899.153589] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3906.076390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3906.330293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.913692] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3914.547647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3922.600932] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3933.153421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.164078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.186881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3935.203265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3935.212615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3935.225322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.245684] Lustre: server umount lustre-OST0000 complete [ 3943.909442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3943.923988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.953681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.972270] Lustre: Skipped 1 previous similar message [ 3950.563543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3950.569801] Lustre: Skipped 2 previous similar messages [ 3955.682707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3955.692950] Lustre: Skipped 1 previous similar message [ 3956.192308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3956.397762] Lustre: server umount lustre-MDT0000 complete [ 3959.785215] LustreError: 128426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.801964] LustreError: 128426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3960.733586] LustreError: 127347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788513887 with bad export cookie 120086880752100502 [ 3960.741911] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3961.133872] Lustre: server umount lustre-MDT0001 complete [ 3973.150506] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3976.161461] Key type lgssc unregistered [ 3976.535034] LNet: 130955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3976.542451] LNetError: 130955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3976.560786] LNet: Removed LNI 192.168.203.151@tcp [ 3977.400834] Key type .llcrypt unregistered [ 3977.404124] Key type ._llcrypt unregistered [ 4002.873289] Key type ._llcrypt registered [ 4002.876445] Key type .llcrypt registered [ 4003.039393] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 4018.210544] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4019.142968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4019.169543] alg: No test for adler32 (adler32-zlib) [ 4020.208348] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 4020.408181] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 4022.049394] Key type lgssc registered [ 4023.036723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4055.954574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4056.004680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.286446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4057.314257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4057.441350] Lustre: lustre-MDT0000: new disk, initializing [ 4057.544393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.563527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4062.361244] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4071.862623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.025187] Lustre: 134355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4072.096417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4072.099155] Lustre: Skipped 1 previous similar message [ 4072.224656] Lustre: lustre-MDT0001: new disk, initializing [ 4072.330645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4072.384972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4072.428726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4076.810540] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4085.358501] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4091.400843] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4097.719296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4097.950679] Lustre: lustre-OST0000: new disk, initializing [ 4097.953689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4097.958075] Lustre: 136321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4098.036206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.201850] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4105.260032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4105.267721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4105.381579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4112.458076] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4117.583494] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4117.765574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4122.137499] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4122.389540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4125.669429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4125.679474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4125.700717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4126.187798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4126.193506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4126.214895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4126.225457] Lustre: Skipped 1 previous similar message [ 4130.566966] Lustre: server umount lustre-OST0000 complete [ 4136.418762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.439371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4136.446467] Lustre: Skipped 1 previous similar message [ 4140.567350] Lustre: server umount lustre-MDT0000 complete [ 4144.022767] LustreError: 134345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788514070 with bad export cookie 10741871293805829191 [ 4144.029131] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4144.036442] LustreError: 134345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4144.393479] Lustre: server umount lustre-MDT0001 complete [ 4153.225643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.732996] LustreError: 138540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.760422] LustreError: 138540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4153.815924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.489796] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4158.945336] LustreError: 138541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.067200] LustreError: 138540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.038440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.342619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4170.272545] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4178.330876] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4184.466928] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4191.481654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4191.746151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4197.087246] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4204.556523] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4209.002856] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4209.227597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4213.610341] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4213.833616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4222.425513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4222.638885] Lustre: lustre-OST0001: new disk, initializing [ 4222.643361] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4222.658408] Lustre: 141909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4222.796458] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4229.103162] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4232.794469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4232.811711] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4232.911125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4237.540367] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4237.761741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4242.637449] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4242.833391] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4248.970334] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89c990f4d800.ost_server_uuid 50 [ 4250.757282] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89c990f4d800.ost_server_uuid in FULL state after 0 sec [ 4258.793670] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4258.801660] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.811626] LustreError: Skipped 1 previous similar message [ 4258.829392] Lustre: Skipped 1 previous similar message [ 4258.834059] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4258.848832] Lustre: Skipped 1 previous similar message [ 4263.912646] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4263.924944] Lustre: Skipped 1 previous similar message [ 4264.540212] Lustre: server umount lustre-OST0001 complete [ 4269.029855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.032855] LustreError: 140363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.052309] Lustre: Skipped 2 previous similar messages [ 4269.064858] LustreError: 140363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4274.156547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.158360] LustreError: 142394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.166448] Lustre: Skipped 3 previous similar messages [ 4274.196366] LustreError: 142394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4282.337754] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4282.487906] Lustre: server umount lustre-OST0000 complete [ 4289.505231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4289.515484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.543426] Lustre: Skipped 1 previous similar message [ 4294.627778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4294.635919] Lustre: Skipped 5 previous similar messages [ 4300.256157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4300.469071] Lustre: server umount lustre-MDT0000 complete [ 4304.530702] LustreError: 138522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788514231 with bad export cookie 10741871293805830073 [ 4304.532239] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4304.545383] LustreError: 138522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4304.965432] Lustre: server umount lustre-MDT0001 complete [ 4326.162200] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 4329.247313] Key type lgssc unregistered [ 4329.626552] LNet: 144914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4329.641612] LNetError: 144914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4329.672891] LNet: Removed LNI 192.168.203.151@tcp [ 4330.649316] Key type .llcrypt unregistered [ 4330.651802] Key type ._llcrypt unregistered [ 4372.433211] Key type ._llcrypt registered [ 4372.436502] Key type .llcrypt registered [ 4372.553222] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 4388.830720] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4390.049705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4390.236060] alg: No test for adler32 (adler32-zlib) [ 4391.308777] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 4391.627759] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 4393.360314] Key type lgssc registered [ 4394.621490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4432.216884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4432.276281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.696605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4433.737628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4433.826472] Lustre: lustre-MDT0000: new disk, initializing [ 4433.919454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4433.936920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4438.302819] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4449.156641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4449.267811] Lustre: 149080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4449.322750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4449.326590] Lustre: Skipped 1 previous similar message [ 4449.391893] Lustre: lustre-MDT0001: new disk, initializing [ 4449.511553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4449.549027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4449.567368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4453.945686] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4464.019851] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4470.432335] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4478.322803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4478.712587] Lustre: lustre-OST0000: new disk, initializing [ 4478.716284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4478.725228] Lustre: 151050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4478.828546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4484.656854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4484.674654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4484.825600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4486.218434] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4497.288379] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4502.553647] Lustre: Failing over lustre-OST0000 [ 4502.716453] Lustre: server umount lustre-OST0000 complete [ 4520.928282] Lustre: 147304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788514431/real 1788514431] req@00000000af0e2ccd x1875393189545088/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788514447 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4520.950765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.868847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4521.953376] Lustre: 147301:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788514432/real 1788514432] req@00000000e6fb1698 x1875393189545216/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788514448 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4521.983780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4522.253686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.265750] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4523.040219] Lustre: 147302:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788514432/real 1788514432] req@0000000043c1607d x1875393189545856/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788514448 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4525.222608] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4527.056109] Lustre: 147304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788514437/real 1788514437] req@00000000d7570947 x1875393189547008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788514453 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4528.923356] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4531.401919] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4531.402436] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4532.448134] Lustre: 147303:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788514442/real 1788514442] req@0000000078bcd9c7 x1875393189548160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788514458 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4532.473272] Lustre: 147303:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4533.161609] Lustre: Failing over lustre-MDT0000 [ 4533.426617] Lustre: server umount lustre-MDT0000 complete [ 4535.690195] LustreError: 149086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4535.700882] LustreError: 149086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4536.804247] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.818281] Lustre: Skipped 2 previous similar messages [ 4536.819213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4536.826292] LustreError: 150173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.778664] LustreError: 149086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.803483] LustreError: 149086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4545.908434] LustreError: 149084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.942428] LustreError: 149084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4551.020600] LustreError: 150172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.068694] LustreError: 150172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4553.190179] Lustre: 147304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788514463/real 1788514463] req@0000000039dd07a5 x1875393189559552/t0(0) o400->MGC192.168.203.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788514479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4553.235746] Lustre: 147304:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4553.251558] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4554.102752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.261128] LustreError: 149085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.293891] LustreError: 149085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 4563.439170] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd0939393ebf620cf [ 4563.456080] Lustre: MGC192.168.203.151@tcp: Connection restored to 0@lo (at 0@lo) [ 4563.469772] Lustre: Skipped 1 previous similar message [ 4563.722834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4563.775804] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4566.379536] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4568.617652] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4569.067441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4569.094071] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4569.155063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4577.768906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4577.784685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4577.809383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4579.301102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4579.308493] Lustre: Skipped 1 previous similar message [ 4584.030587] Lustre: server umount lustre-OST0000 complete [ 4589.540566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4589.540881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.548706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4589.566329] Lustre: Skipped 2 previous similar messages [ 4594.518259] Lustre: server umount lustre-MDT0000 complete [ 4594.660647] LustreError: 150173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.681150] LustreError: 150173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4598.359448] LustreError: 151048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788514524 with bad export cookie 15029518644993728719 [ 4598.362717] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4598.375644] LustreError: 151048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4598.837238] Lustre: server umount lustre-MDT0001 complete [ 4610.184974] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 4613.048262] Key type lgssc unregistered [ 4613.311566] LNet: 154703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4613.325071] LNetError: 154703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4613.350888] LNet: Removed LNI 192.168.203.151@tcp [ 4614.254490] Key type .llcrypt unregistered [ 4614.257822] Key type ._llcrypt unregistered [ 4656.219322] Key type ._llcrypt registered [ 4656.222333] Key type .llcrypt registered [ 4656.336896] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4657.561463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4657.696070] alg: No test for adler32 (adler32-zlib) [ 4658.885764] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 4659.190638] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 4660.936309] Key type lgssc registered [ 4662.356569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4670.552759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4670.571661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.078199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.914746] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4682.213378] LustreError: 156224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.226962] LustreError: 156224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4687.336542] LustreError: 156225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.343361] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4692.309420] LustreError: 156253:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4692.321530] LustreError: 156253:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4692.330715] LustreError: 156253:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4693.043240] Lustre: server umount lustre-MDT0000 complete [ 4712.020809] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 4724.250463] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4763.868444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.084493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4764.116446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4764.200227] Lustre: lustre-MDT0000: new disk, initializing [ 4764.268205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4764.282472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4769.235583] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4780.202273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.378418] Lustre: 159561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4780.456254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4780.466048] Lustre: Skipped 1 previous similar message [ 4780.594549] Lustre: lustre-MDT0001: new disk, initializing [ 4780.717611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4780.752616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4780.771406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4785.963603] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4796.760389] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4804.478373] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4812.346858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4812.841133] Lustre: lustre-OST0000: new disk, initializing [ 4812.849940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4812.859844] Lustre: 161529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4813.003490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.030513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4819.039259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4819.150668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4820.231741] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4831.240263] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4835.474316] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4837.347236] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:39:21 (1788514761) [ 5116.385321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.389299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5116.406356] Lustre: Skipped 2 previous similar messages [ 5116.419700] Lustre: Skipped 1 previous similar message [ 5121.514357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5121.528070] Lustre: Skipped 2 previous similar messages [ 5122.725236] Lustre: server umount lustre-MDT0000 complete [ 5126.628969] LustreError: 159573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5126.663491] LustreError: 159573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5127.121067] LustreError: 159552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788515053 with bad export cookie 8697705679775105496 [ 5127.121714] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5127.128710] LustreError: 159552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5131.748573] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.757616] LustreError: 161123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.781789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5133.645571] Lustre: server umount lustre-MDT0001 complete [ 5152.224327] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5152.468543] Lustre: server umount lustre-OST0000 complete [ 5164.044812] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 5181.524926] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5220.168416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5220.399593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5220.428742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5220.528177] Lustre: lustre-MDT0000: new disk, initializing [ 5220.615794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.631061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5225.479813] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5234.883268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.040191] Lustre: 166218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5235.128301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5235.135714] Lustre: Skipped 1 previous similar message [ 5235.210139] Lustre: lustre-MDT0001: new disk, initializing [ 5235.305232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5235.357401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5235.373733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5239.630427] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5248.429820] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5254.282211] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5260.880800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5261.249314] Lustre: lustre-OST0000: new disk, initializing [ 5261.253241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5261.268666] Lustre: 168186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5261.366461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5267.448496] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5271.142224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5271.157250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5271.358804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5276.772856] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5281.429693] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5281.723601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5286.450498] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5286.647322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5292.004136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5292.008080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.023838] LustreError: Skipped 1 previous similar message [ 5292.042090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5292.052533] Lustre: Skipped 2 previous similar messages [ 5294.924452] Lustre: server umount lustre-OST0000 complete [ 5301.730936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5301.749960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5301.770284] Lustre: Skipped 1 previous similar message [ 5301.791828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5305.533960] Lustre: server umount lustre-MDT0000 complete [ 5307.365604] LustreError: 169526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.390903] LustreError: 169526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5309.434255] LustreError: 166210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788515235 with bad export cookie 8697705679775264326 [ 5309.446987] LustreError: 166210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5309.447566] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5309.758271] Lustre: server umount lustre-MDT0001 complete [ 5318.219825] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:47:22 (1788515242) [ 5335.563514] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 5345.783344] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5379.683251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.912756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5379.946242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5380.025519] Lustre: lustre-MDT0000: new disk, initializing [ 5380.101588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.134519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5383.899889] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5393.688755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.835148] Lustre: 172943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5394.147857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5398.637363] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5408.943989] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5414.668660] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5421.321499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5421.615465] Lustre: lustre-OST0000: new disk, initializing [ 5421.618631] Lustre: Skipped 1 previous similar message [ 5421.628813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5421.635904] Lustre: Skipped 2 previous similar messages [ 5421.643193] Lustre: 174920:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5427.559361] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5429.312387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5429.324961] Lustre: Skipped 1 previous similar message [ 5429.333762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5429.465084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5436.936743] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5445.094120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.105589] Lustre: Skipped 1 previous similar message [ 5445.107951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5445.133916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5445.136806] Lustre: Skipped 2 previous similar messages [ 5450.472892] Lustre: server umount lustre-OST0000 complete [ 5455.329271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.342102] Lustre: Skipped 1 previous similar message [ 5460.450060] LustreError: 173848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5460.481403] LustreError: 173848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5460.528276] Lustre: server umount lustre-MDT0000 complete [ 5464.387884] LustreError: 172935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788515390 with bad export cookie 8697705679775265208 [ 5464.395410] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.404961] LustreError: 172935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5464.884670] Lustre: server umount lustre-MDT0001 complete [ 5474.332506] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:49:58 (1788515398) [ 5492.174966] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 5503.864449] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5543.298810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.665780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5543.692884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5543.743805] Lustre: lustre-MDT0000: new disk, initializing [ 5543.799471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.811377] Lustre: Skipped 2 previous similar messages [ 5543.840495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5549.301426] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5560.330516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5560.473821] Lustre: 179256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5560.790224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5565.712425] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5575.306995] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5581.352221] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5587.492712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5587.736221] Lustre: 181230:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5593.942879] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5597.810762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5597.956336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5604.720415] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5613.538630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.542747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5613.544811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5613.544820] Lustre: Skipped 4 previous similar messages [ 5613.558425] Lustre: Skipped 1 previous similar message [ 5617.905468] Lustre: server umount lustre-OST0000 complete [ 5622.244686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5627.394408] Lustre: server umount lustre-MDT0000 complete [ 5628.898584] LustreError: 179266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5628.942243] LustreError: 179266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5632.383496] LustreError: 179247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788515558 with bad export cookie 8697705679775266293 [ 5632.385580] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5632.419508] LustreError: 179247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5632.935239] Lustre: server umount lustre-MDT0001 complete [ 5648.318679] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 5652.224503] Key type lgssc unregistered [ 5652.628610] LNet: 183085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5652.649042] LNetError: 183085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5652.677508] LNet: Removed LNI 192.168.203.151@tcp [ 5653.955794] Key type .llcrypt unregistered [ 5653.961824] Key type ._llcrypt unregistered [ 5685.331493] Key type ._llcrypt registered [ 5685.333569] Key type .llcrypt registered [ 5685.466837] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5686.530556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5686.569104] alg: No test for adler32 (adler32-zlib) [ 5687.861442] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 5688.173857] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 5689.865580] Key type lgssc registered [ 5690.944989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5699.444563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5699.478801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.006614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.914683] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5711.331026] LustreError: 184501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.357260] LustreError: 184501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5714.927151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5715.295230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5719.836320] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5727.578651] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5733.030728] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5740.334595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5740.645236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5745.774583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5746.807867] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5754.238975] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5776.869528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5776.883206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.914587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5780.961511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5780.963898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5780.965643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.000595] Lustre: Skipped 1 previous similar message [ 5782.812741] Lustre: server umount lustre-OST0000 complete [ 5787.111939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5787.125812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.160992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5791.202132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5791.202805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.163192] Lustre: server umount lustre-MDT0000 complete [ 5796.322979] LustreError: 184500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.341406] LustreError: 184500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5797.243248] LustreError: 184481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788515723 with bad export cookie 16973854186083017663 [ 5797.250230] LustreError: 184481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5797.252663] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5797.604645] Lustre: server umount lustre-MDT0001 complete [ 5808.565303] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 5811.405833] Key type lgssc unregistered [ 5811.759604] LNet: 188056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5811.768938] LNetError: 188056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5811.786781] LNet: Removed LNI 192.168.203.151@tcp [ 5812.460152] Key type .llcrypt unregistered [ 5812.463078] Key type ._llcrypt unregistered [ 5841.293630] Key type ._llcrypt registered [ 5841.299280] Key type .llcrypt registered [ 5841.440863] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5842.960054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5843.015437] alg: No test for adler32 (adler32-zlib) [ 5844.149440] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 5844.439492] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 5846.224600] Key type lgssc registered [ 5847.719894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5856.244746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5856.270190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.976500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5863.063533] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5867.499544] LustreError: 189470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.523305] LustreError: 189470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5870.913970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5871.322629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5875.186366] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5882.841713] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5888.810169] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5895.938481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5896.166582] LustreError: 191290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5896.301489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.717308] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5903.358732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5910.164257] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5916.642431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.651244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.661083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.692258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5918.693892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.700566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5918.711771] Lustre: Skipped 1 previous similar message [ 5922.152731] Lustre: lustre-OST0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 5922.165078] Lustre: Skipped 1 previous similar message [ 5927.293753] Lustre: lustre-OST0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 5927.299293] Lustre: Skipped 2 previous similar messages [ 5929.952196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5930.141172] Lustre: server umount lustre-OST0000 complete [ 5937.135436] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89c982c5e800.ost_server_uuid 50 [ 5938.635851] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5951.286089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.289074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.307821] Lustre: Skipped 1 previous similar message [ 5951.338371] Lustre: Skipped 3 previous similar messages [ 5957.571819] Lustre: server umount lustre-MDT0000 complete [ 5958.626205] LustreError: 189470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.636079] LustreError: 189470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5960.785761] LustreError: 190204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788515887 with bad export cookie 14787617312114263035 [ 5960.792847] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5960.804032] LustreError: 190204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5961.062739] Lustre: server umount lustre-MDT0001 complete [ 5968.320079] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:58:12 (1788515892) [ 5974.495229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.913654] LustreError: 193384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5974.983916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.767414] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5980.134085] LustreError: 193385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5986.926965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5987.204126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5991.384587] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5999.452702] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6006.284590] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6013.215775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6013.604845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.755785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6020.239766] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6027.077667] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6034.694506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6034.972415] Lustre: lustre-OST0001: new disk, initializing [ 6034.975869] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6034.987258] Lustre: 196228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6035.123444] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6041.687301] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6043.234861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6043.245449] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6043.436318] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6052.571475] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6055.905586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6055.910921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.935047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6055.949595] Lustre: Skipped 2 previous similar messages [ 6057.958357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.240152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6070.546869] Lustre: server umount lustre-OST0000 complete [ 6073.707885] LustreError: 195208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.743917] LustreError: 195208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6077.544658] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6077.768440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6083.108462] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6083.402233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6089.710660] LustreError: 196667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.736390] LustreError: 196667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6099.944468] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6099.944868] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.972803] Lustre: Skipped 1 previous similar message [ 6099.995154] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6099.997798] Lustre: Skipped 10 previous similar messages [ 6103.602714] Lustre: server umount lustre-OST0001 complete [ 6110.177955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.179609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6110.195806] Lustre: Skipped 2 previous similar messages [ 6110.220089] LustreError: Skipped 1 previous similar message [ 6114.325398] Lustre: server umount lustre-MDT0000 complete [ 6118.427962] LustreError: 193366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788516044 with bad export cookie 14787617312114264106 [ 6118.437960] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6118.449267] LustreError: 193366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6118.834731] Lustre: server umount lustre-MDT0001 complete [ 6142.829082] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:01:07 (1788516067) [ 6149.479602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.928516] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6149.983461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6150.266771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6154.469646] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6162.589121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.644784] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6162.674773] Lustre: 199682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6167.425366] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6176.973918] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6184.246609] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6192.420156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6192.787642] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6198.913802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6199.539452] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6208.224809] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6215.930678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6216.051190] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6216.379793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6216.389515] Lustre: Skipped 2 previous similar messages [ 6224.133270] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6232.422544] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6237.678568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.686731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6237.711706] Lustre: Skipped 1 previous similar message [ 6237.729922] Lustre: Skipped 3 previous similar messages [ 6241.789483] Lustre: server umount lustre-OST0000 complete [ 6242.789644] LustreError: 201530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.802290] LustreError: 201530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6258.145183] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6258.160102] LustreError: Skipped 1 previous similar message [ 6264.157723] Lustre: server umount lustre-OST0001 complete [ 6270.434240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.464670] Lustre: Skipped 2 previous similar messages [ 6274.912552] Lustre: server umount lustre-MDT0000 complete [ 6279.282528] LustreError: 199675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788516205 with bad export cookie 14787617312114265513 [ 6279.292588] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6279.294055] LustreError: 199675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6279.631304] Lustre: server umount lustre-MDT0001 complete [ 6300.631610] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:03:45 (1788516225) [ 6314.616179] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 6324.495695] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6359.713551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.086509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6360.162562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6360.335359] Lustre: lustre-MDT0000: new disk, initializing [ 6360.511629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.531342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6366.379913] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6378.834249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6379.083929] Lustre: 207784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6379.156443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6379.160347] Lustre: Skipped 1 previous similar message [ 6379.306780] Lustre: lustre-MDT0001: new disk, initializing [ 6379.505114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6379.514085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6384.196000] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6394.883546] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6401.731633] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6408.969891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6409.342535] Lustre: lustre-OST0000: new disk, initializing [ 6409.348574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6409.358573] Lustre: 209751:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6416.988116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6417.012372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6417.232779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6417.283521] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6427.664659] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6433.655754] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6433.966624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6438.940403] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6439.248232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6442.989681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6442.990180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6442.999915] LustreError: Skipped 2 previous similar messages [ 6443.002479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6443.007934] Lustre: Skipped 2 previous similar messages [ 6443.023657] Lustre: Skipped 9 previous similar messages [ 6447.723286] Lustre: server umount lustre-OST0000 complete [ 6458.288312] Lustre: server umount lustre-MDT0000 complete [ 6458.338214] LustreError: 207789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.378423] LustreError: 207789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6462.719921] LustreError: 209750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788516389 with bad export cookie 14787617312114267067 [ 6462.737429] LustreError: 209750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6462.742128] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.294835] Lustre: server umount lustre-MDT0001 complete [ 6472.954325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.376348] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6487.380935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6493.216216] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6502.040359] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6508.524544] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6517.219794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6524.729846] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6533.935188] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6538.455784] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6538.637260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6543.383794] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6543.622672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6548.461876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6551.544565] Lustre: server umount lustre-OST0000 complete [ 6558.277850] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6562.357734] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6592.705388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6598.821498] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6606.350319] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6609.052822] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6613.556498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.566482] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.566631] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6613.620567] Lustre: Skipped 1 previous similar message [ 6618.594026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.609948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6618.626012] Lustre: Skipped 5 previous similar messages [ 6618.645858] Lustre: Skipped 5 previous similar messages [ 6620.695472] Lustre: server umount lustre-OST0000 complete [ 6640.096104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6640.905386] Lustre: server umount lustre-MDT0000 complete [ 6644.197194] LustreError: 211968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.233894] LustreError: 211968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6645.323680] LustreError: 211951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788516571 with bad export cookie 14787617312114267956 [ 6645.338573] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6645.349599] LustreError: 211951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6645.742690] Lustre: server umount lustre-MDT0001 complete [ 6656.463669] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:09:40 (1788516580) [ 6663.484139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6664.012242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6664.017983] Lustre: Skipped 6 previous similar messages [ 6668.306659] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6676.586679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6681.464648] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6688.002207] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6693.638558] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6700.692919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.070180] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6716.184485] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6720.679497] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6720.898402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6725.592338] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6725.879570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6732.830620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6733.048571] Lustre: lustre-OST0001: new disk, initializing [ 6733.051790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6733.059363] Lustre: 221305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6739.055524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6739.065330] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6739.203178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6739.504955] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6749.525463] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6754.607143] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6754.884451] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6759.031782] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6759.312843] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6765.031068] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6765.040642] LustreError: Skipped 2 previous similar messages [ 6767.567101] Lustre: server umount lustre-OST0001 complete [ 6774.139505] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6774.360592] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6779.704218] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6779.972835] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6809.130852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6814.700617] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6814.714385] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6814.719430] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6814.741722] Lustre: Skipped 1 previous similar message [ 6816.079671] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6824.906441] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6826.835332] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6843.360349] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6843.797576] Lustre: server umount lustre-OST0001 complete [ 6876.133285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6876.146488] Lustre: Skipped 20 previous similar messages [ 6878.176997] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6878.485031] Lustre: server umount lustre-OST0000 complete [ 6886.383095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.391897] Lustre: Skipped 9 previous similar messages [ 6888.898439] Lustre: server umount lustre-MDT0000 complete [ 6892.726597] LustreError: 217916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788516819 with bad export cookie 14787617312114268922 [ 6892.735582] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6892.739176] LustreError: 217916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6907.360243] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6907.843693] Lustre: server umount lustre-MDT0001 complete [ 6933.018898] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:14:17 (1788516857) [ 6940.151362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.579598] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6940.671126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6946.547668] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6955.842732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6955.994299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6956.050436] Lustre: 226554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6961.138712] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6969.825437] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6975.672069] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6982.954659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6983.210534] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6990.332452] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6998.306680] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7007.709804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7007.831620] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7015.494799] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7023.210059] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.948696] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7029.741559] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7029.972906] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7034.862312] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7035.072907] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7041.011455] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89c992605000.ost_server_uuid 50 [ 7042.627127] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89c992605000.ost_server_uuid in IDLE state after 0 sec [ 7043.632857] Lustre: Permanently deactivating lustre-OST0001 [ 7043.644623] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7046.987748] Lustre: Permanently reactivating lustre-OST0001 [ 7051.747389] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7051.757987] LustreError: Skipped 3 previous similar messages [ 7057.062561] Lustre: server umount lustre-OST0001 complete [ 7061.986371] LustreError: 228402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7062.001313] LustreError: 228402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7067.178901] Lustre: server umount lustre-OST0000 complete [ 7078.636837] Lustre: server umount lustre-MDT0000 complete [ 7082.493436] LustreError: 226548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788517009 with bad export cookie 14787617312114270210 [ 7082.504792] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7082.506020] LustreError: 226548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7082.964427] Lustre: server umount lustre-MDT0001 complete [ 7107.625399] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:17:11 (1788517031) [ 7114.643033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.879105] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7114.914582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7114.923576] Lustre: Skipped 5 previous similar messages [ 7119.470432] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7127.651110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.783890] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7127.829511] Lustre: 233247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7132.128388] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7140.949145] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7146.480957] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7153.529302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7153.843139] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7153.874510] Lustre: Permanently deactivating lustre-OST0000 [ 7153.895477] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7159.340225] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7160.842531] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7167.910785] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7174.012815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7174.088933] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7179.834247] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7181.492966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7187.270693] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7189.600873] Lustre: Permanently reactivating lustre-OST0000 [ 7189.630452] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7189.638040] Lustre: Skipped 2 previous similar messages [ 7196.258233] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7196.269546] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 000000002018b84d, cur 1788517122 deadline 1788517115 last 1788517085 [ 7196.358717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7202.318984] LustreError: 233241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.51@tcp arrived at 1788517128 with bad export cookie 14787617312114273332 [ 7211.308330] Lustre: server umount lustre-OST0001 complete [ 7223.266812] Lustre: server umount lustre-OST0000 complete [ 7234.892888] Lustre: server umount lustre-MDT0000 complete [ 7238.764846] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7239.238208] Lustre: server umount lustre-MDT0001 complete [ 7250.971113] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7253.666764] Key type lgssc unregistered [ 7254.062663] LNet: 238130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7254.073174] LNetError: 238130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7254.102377] LNet: Removed LNI 192.168.203.151@tcp [ 7255.053759] Key type .llcrypt unregistered [ 7255.055846] Key type ._llcrypt unregistered [ 7277.989602] Key type ._llcrypt registered [ 7277.995423] Key type .llcrypt registered [ 7278.111663] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7279.328684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7279.750462] alg: No test for adler32 (adler32-zlib) [ 7280.878769] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 7281.095183] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7282.752157] Key type lgssc registered [ 7284.334266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7295.312669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7295.341810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.832969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7301.053987] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7307.241429] LustreError: 239594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.270880] LustreError: 239594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7308.822246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.962474] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7308.987941] Lustre: 239583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7308.998200] Lustre: Permanently deactivating lustre-MDT0001 [ 7309.009359] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7309.321566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7309.854944] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7309.871147] LustreError: 239625:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7309.894986] LustreError: 239625:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7309.914652] LustreError: 239625:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 7314.050410] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7320.332353] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7325.648906] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7331.823114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.195686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7334.249430] LustreError: 241435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7334.269139] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7338.123977] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7339.503152] LustreError: 241436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7341.552112] LustreError: 241435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7341.581858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7345.250340] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7346.663537] LustreError: 241948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.690150] LustreError: 241948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7352.486269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7352.703298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7357.949139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7357.967600] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7358.618356] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7365.635542] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7368.508519] Lustre: Permanently reactivating lustre-MDT0001 [ 7368.526285] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7368.531700] Lustre: Skipped 2 previous similar messages [ 7374.219981] Lustre: lustre-MDT0001: Client 225b3c75-3990-4af1-905c-831cd0b0ac23 (at 192.168.203.51@tcp) reconnecting [ 7379.795483] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7381.074765] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7386.673267] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7386.892219] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7391.273436] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7391.500379] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7393.805448] Lustre: Permanently deactivating lustre-MDT0001 [ 7393.826783] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7393.835366] Lustre: Skipped 2 previous similar messages [ 7399.533060] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7406.573393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7406.584108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.596855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7408.512938] Lustre: server umount lustre-MDT0000 complete [ 7409.126247] LustreError: 239595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7409.131556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7409.147332] LustreError: 239595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7409.177061] Lustre: Skipped 2 previous similar messages [ 7411.929749] LustreError: 239576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788517338 with bad export cookie 11899621927544486587 [ 7411.940247] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7411.943068] LustreError: 239576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7412.268272] Lustre: server umount lustre-MDT0001 complete [ 7415.509135] Lustre: server umount lustre-OST0000 complete [ 7424.633545] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:22:29 (1788517349) [ 7434.515316] Lustre: server umount lustre-OST0001 complete [ 7441.029101] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7450.131503] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7481.885255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.054371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7482.061168] Lustre: Skipped 2 previous similar messages [ 7482.097237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7482.186648] Lustre: lustre-MDT0000: new disk, initializing [ 7482.300225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.322622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7487.648308] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7498.788167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7498.902307] Lustre: 247853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7498.957700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7498.961256] Lustre: Skipped 1 previous similar message [ 7499.033693] Lustre: lustre-MDT0001: new disk, initializing [ 7499.122663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7499.163136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7499.175421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7503.667844] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7512.429164] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7517.248156] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7524.250647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.499771] Lustre: lustre-OST0000: new disk, initializing [ 7524.504859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7524.512041] Lustre: 249822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7524.606882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7530.050455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7530.068515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7530.203020] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7531.269096] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7540.912212] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7543.432946] LustreError: 247859:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7552.318386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7552.543065] Lustre: lustre-OST0001: new disk, initializing [ 7552.548238] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7552.554666] Lustre: 250935:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7553.456278] LustreError: 247859:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7553.474836] LustreError: 249075:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7558.285023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7558.295244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7558.439671] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7559.709450] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7563.528113] LustreError: 249075:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7570.742950] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7572.227593] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7579.112877] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7579.122614] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7579.138777] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7579.152676] Lustre: Skipped 1 previous similar message [ 7579.166848] Lustre: Skipped 1 previous similar message [ 7584.230621] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7584.243471] Lustre: Skipped 1 previous similar message [ 7588.320136] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7588.494970] Lustre: server umount lustre-OST0001 complete [ 7589.371130] LustreError: 249804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7589.390935] LustreError: 249804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7607.269080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7607.275625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.284813] LustreError: Skipped 2 previous similar messages [ 7607.288593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7607.311159] Lustre: Skipped 1 previous similar message [ 7612.918402] Lustre: server umount lustre-OST0000 complete [ 7616.999293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7617.013810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7617.023956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7617.029119] Lustre: Skipped 3 previous similar messages [ 7625.185252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7625.196722] Lustre: Skipped 3 previous similar messages [ 7631.328470] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7631.597055] Lustre: server umount lustre-MDT0000 complete [ 7635.388580] LustreError: 247844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788517561 with bad export cookie 11899621927544489401 [ 7635.398265] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.401283] LustreError: 247844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7641.978808] Lustre: server umount lustre-MDT0001 complete [ 7652.193889] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7654.896612] Key type lgssc unregistered [ 7655.215846] LNet: 252972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7655.225300] LNetError: 252972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7655.244555] LNet: Removed LNI 192.168.203.151@tcp [ 7656.174419] Key type .llcrypt unregistered [ 7656.177981] Key type ._llcrypt unregistered [ 7695.149842] Key type ._llcrypt registered [ 7695.159149] Key type .llcrypt registered [ 7695.339562] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7696.724404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7696.812688] alg: No test for adler32 (adler32-zlib) [ 7697.995476] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 7698.224902] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7699.912347] Key type lgssc registered [ 7701.237290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7710.413874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7710.461301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.794824] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7711.833180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7712.062800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7718.292419] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7729.941656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7730.160165] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7730.247287] Lustre: 255176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7730.834756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7737.845992] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7746.060712] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7752.349539] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7759.165698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7759.503714] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7759.818323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.062880] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7767.702439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7775.618850] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7834.081343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7834.100846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.124081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7834.593928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7834.604576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.928623] Lustre: server umount lustre-OST0000 complete [ 7840.110990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7853.450695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7853.853483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7854.898121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7854.920462] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7854.921640] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7854.945612] Lustre: Skipped 1 previous similar message [ 7860.754481] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7869.040321] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7911.396829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.401795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7911.426112] Lustre: Skipped 1 previous similar message [ 7911.448946] Lustre: Skipped 2 previous similar messages [ 7915.881153] Lustre: server umount lustre-OST0000 complete [ 7921.633439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7921.635479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7921.651144] Lustre: Skipped 1 previous similar message [ 7926.881650] Lustre: server umount lustre-MDT0000 complete [ 7930.863921] LustreError: 255168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788517857 with bad export cookie 7909530755305071369 [ 7930.874975] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7930.880313] LustreError: 255168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7931.197727] Lustre: server umount lustre-MDT0001 complete [ 7941.883541] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7945.186122] Key type lgssc unregistered [ 7945.569811] LNet: 260599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7945.584909] LNetError: 260599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7945.607509] LNet: Removed LNI 192.168.203.151@tcp [ 7946.799248] Key type .llcrypt unregistered [ 7946.804661] Key type ._llcrypt unregistered [ 7972.187400] Key type ._llcrypt registered [ 7972.190728] Key type .llcrypt registered [ 7972.289812] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7973.457207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7973.509141] alg: No test for adler32 (adler32-zlib) [ 7974.684473] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 7975.023980] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7976.767078] Key type lgssc registered [ 7977.947690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7986.371787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7986.408230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.024376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.423946] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7998.436869] LustreError: 262010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.460657] LustreError: 262010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8001.092348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.394990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8005.776568] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8013.104276] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8018.736613] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8025.092684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8025.479616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8031.862141] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8032.636922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8039.764400] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8063.464746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8063.490264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8063.493347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8068.584296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8068.595733] Lustre: Skipped 1 previous similar message [ 8068.771204] Lustre: server umount lustre-OST0000 complete [ 8073.698475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8073.722201] Lustre: Skipped 1 previous similar message [ 8073.731890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8073.746553] Lustre: Skipped 1 previous similar message [ 8078.305768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8078.317070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8078.337605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8079.972695] Lustre: server umount lustre-MDT0000 complete [ 8083.939053] LustreError: 264501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.960927] LustreError: 264501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8084.607362] LustreError: 265563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788518011 with bad export cookie 13413735604062910227 [ 8084.617309] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8084.620037] LustreError: 265563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8085.125372] Lustre: server umount lustre-MDT0001 complete [ 8096.411330] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8099.660914] Key type lgssc unregistered [ 8100.063599] LNet: 266316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8100.077349] LNetError: 266316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8100.116351] LNet: Removed LNI 192.168.203.151@tcp [ 8101.164808] Key type .llcrypt unregistered [ 8101.168381] Key type ._llcrypt unregistered [ 8118.566065] Key type ._llcrypt registered [ 8118.568556] Key type .llcrypt registered [ 8118.671720] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8120.119790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8120.160156] alg: No test for adler32 (adler32-zlib) [ 8121.176995] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 8121.430155] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8123.088474] Key type lgssc registered [ 8124.359778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8132.366386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8132.405903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8134.022751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8139.020308] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8144.360471] LustreError: 267534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8144.390083] LustreError: 267534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8148.104137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8148.496332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8153.295406] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8160.510780] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8168.228377] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8175.466920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8175.953093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8179.002576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8182.966873] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8189.967410] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8208.868880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.880947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8208.895608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8208.911195] Lustre: Skipped 1 previous similar message [ 8211.382692] Lustre: server umount lustre-OST0000 complete [ 8219.119889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8219.124397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8219.142105] Lustre: Skipped 1 previous similar message [ 8219.168695] Lustre: Skipped 2 previous similar messages [ 8224.226807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8224.230439] Lustre: Skipped 1 previous similar message [ 8229.346570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8229.362277] Lustre: Skipped 1 previous similar message [ 8230.368346] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8230.717943] Lustre: server umount lustre-MDT0000 complete [ 8234.467532] LustreError: 267530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8234.479243] LustreError: 267530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8235.728549] LustreError: 267514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788518162 with bad export cookie 10281728589075761025 [ 8235.740665] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8236.448352] Lustre: server umount lustre-MDT0001 complete [ 8250.107732] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8253.100699] Key type lgssc unregistered [ 8253.436530] LNet: 271212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8253.440979] LNetError: 271212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8253.455107] LNet: Removed LNI 192.168.203.151@tcp [ 8254.557401] Key type .llcrypt unregistered [ 8254.564852] Key type ._llcrypt unregistered [ 8280.144659] Key type ._llcrypt registered [ 8280.147551] Key type .llcrypt registered [ 8280.282897] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8281.360425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8281.396594] alg: No test for adler32 (adler32-zlib) [ 8282.459448] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 8282.780821] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8284.536122] Key type lgssc registered [ 8286.167897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8294.748425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8294.775616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8296.238885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8300.456456] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8305.638138] LustreError: 272628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8305.667693] LustreError: 272628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8308.606054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.069859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8313.012220] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8319.753930] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8325.235703] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8331.822843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8332.226389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.191881] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8341.457318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8346.060861] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8372.197083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8372.204498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8372.205785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8372.216834] Lustre: Skipped 1 previous similar message [ 8372.232316] Lustre: Skipped 1 previous similar message [ 8377.316219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8377.340085] Lustre: Skipped 1 previous similar message [ 8378.084964] Lustre: server umount lustre-OST0000 complete [ 8386.018306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8386.047085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8386.082375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8387.583407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8388.988799] Lustre: server umount lustre-MDT0000 complete [ 8392.685089] LustreError: 272625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8392.728590] LustreError: 272625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8393.840041] LustreError: 275981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788518320 with bad export cookie 7325321599037062817 [ 8393.850279] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8393.852385] LustreError: 275981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8394.169868] Lustre: server umount lustre-MDT0001 complete [ 8405.390530] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8408.325991] Key type lgssc unregistered [ 8408.592553] LNet: 276933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8408.610606] LNetError: 276933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8408.651978] LNet: Removed LNI 192.168.203.151@tcp [ 8409.504809] Key type .llcrypt unregistered [ 8409.506792] Key type ._llcrypt unregistered [ 8425.800698] Key type ._llcrypt registered [ 8425.804546] Key type .llcrypt registered [ 8425.954047] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8427.231899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8427.374038] alg: No test for adler32 (adler32-zlib) [ 8428.509878] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 8428.744341] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8430.416161] Key type lgssc registered [ 8431.735290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8439.734203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8439.750165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8441.344068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8445.909477] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8450.534566] LustreError: 278164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.569144] LustreError: 278164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8454.144759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.576908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8459.316498] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8467.170918] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8474.161765] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8481.357556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8481.705341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.807951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8488.428425] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8497.317988] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8514.534147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8514.544553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.552199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8514.590059] Lustre: Skipped 1 previous similar message [ 8519.063731] Lustre: server umount lustre-OST0000 complete [ 8524.775502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.778922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.789680] Lustre: Skipped 1 previous similar message [ 8524.803347] Lustre: Skipped 2 previous similar messages [ 8529.892374] LustreError: 278164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.905264] LustreError: 278164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8529.995633] Lustre: server umount lustre-MDT0000 complete [ 8534.349548] LustreError: 278131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788518460 with bad export cookie 9886822073212925891 [ 8534.359457] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8534.364045] LustreError: 278131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8534.747940] Lustre: server umount lustre-MDT0001 complete [ 8548.068883] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8550.750235] Key type lgssc unregistered [ 8551.032594] LNet: 281868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8551.048417] LNetError: 281868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8551.078661] LNet: Removed LNI 192.168.203.151@tcp [ 8551.890884] Key type .llcrypt unregistered [ 8551.892497] Key type ._llcrypt unregistered [ 8563.446898] Key type ._llcrypt registered [ 8563.450392] Key type .llcrypt registered [ 8563.579338] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8580.938242] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8595.956905] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8597.481826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8597.519391] alg: No test for adler32 (adler32-zlib) [ 8598.614255] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 8598.877388] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8600.576177] Key type lgssc registered [ 8601.929374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8637.245429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8637.289834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8638.626543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8638.649212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8638.806425] Lustre: lustre-MDT0000: new disk, initializing [ 8638.913624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8638.945665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8643.428186] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8653.565126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.715898] Lustre: 285418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8653.811319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8653.821557] Lustre: Skipped 1 previous similar message [ 8653.920238] Lustre: lustre-MDT0001: new disk, initializing [ 8654.031491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8654.090625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8654.124162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8659.068116] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8670.006589] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8677.068440] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8684.455174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8684.844285] Lustre: lustre-OST0000: new disk, initializing [ 8684.847485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8684.852268] Lustre: 287384:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8684.963436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8691.380030] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8692.351209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8692.361611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8692.482294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8701.567514] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8706.650081] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8706.908738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8711.564820] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8711.820403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8718.309508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8718.310321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.324800] LustreError: Skipped 1 previous similar message [ 8718.328883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8718.362695] Lustre: Skipped 1 previous similar message [ 8720.446252] Lustre: server umount lustre-OST0000 complete [ 8725.986219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8725.998188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8726.017274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8726.023765] Lustre: Skipped 1 previous similar message [ 8728.545922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8728.547222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8728.585577] Lustre: Skipped 1 previous similar message [ 8730.903244] Lustre: server umount lustre-MDT0000 complete [ 8734.731953] LustreError: 285408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788518661 with bad export cookie 1311296715173539094 [ 8734.741137] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8734.748717] LustreError: 285408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8735.145343] Lustre: server umount lustre-MDT0001 complete [ 8743.920225] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:44:27 (1788518667) [ 8750.432441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.897381] LustreError: 289798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.916557] LustreError: 289798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8750.976831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.416637] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8756.194705] LustreError: 289799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.331356] LustreError: 289798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8763.926916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.173630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8768.144291] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8775.323290] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8779.968870] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8785.452253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8785.772408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8791.411743] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8798.544914] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8809.443552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8809.458689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.483943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8811.493761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8814.412943] Lustre: server umount lustre-OST0000 complete [ 8820.706265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8820.730863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8820.765615] Lustre: Skipped 1 previous similar message [ 8820.775042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8820.794640] Lustre: Skipped 2 previous similar messages [ 8824.056577] Lustre: server umount lustre-MDT0000 complete [ 8826.849791] LustreError: 292291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8826.876469] LustreError: 292291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8827.700222] LustreError: 289778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788518754 with bad export cookie 1311296715173539976 [ 8827.701323] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8827.707462] LustreError: 289778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8828.136435] Lustre: server umount lustre-MDT0001 complete [ 8840.296612] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8843.239453] Key type lgssc unregistered [ 8843.614961] LNet: 293333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8843.623215] LNetError: 293333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8844.647292] LNet: Removed LNI 192.168.203.151@tcp [ 8845.570504] Key type .llcrypt unregistered [ 8845.572748] Key type ._llcrypt unregistered [ 8874.048536] Key type ._llcrypt registered [ 8874.051291] Key type .llcrypt registered [ 8874.193768] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8890.079437] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8891.024045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8891.050462] alg: No test for adler32 (adler32-zlib) [ 8892.355853] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 8892.567816] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8894.225558] Key type lgssc registered [ 8895.666405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8943.652558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8943.714492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8944.952961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8945.007272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8945.070372] Lustre: lustre-MDT0000: new disk, initializing [ 8945.126079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.141798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8949.903505] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8962.073532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.210072] Lustre: 297155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8962.299481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8962.305895] Lustre: Skipped 1 previous similar message [ 8962.414187] Lustre: lustre-MDT0001: new disk, initializing [ 8962.516396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8962.564374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8962.580171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8968.121759] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8979.782472] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8986.906535] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8995.051830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8995.414559] Lustre: lustre-OST2710: new disk, initializing [ 8995.421344] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8995.435319] Lustre: 299125:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8995.566446] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9002.763946] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9003.596043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9003.612521] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9003.745031] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9013.479539] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9020.675545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9020.859219] Lustre: lustre-OST03e8: new disk, initializing [ 9020.861996] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9020.866878] Lustre: 300188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9020.962220] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9027.365637] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9029.225314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9029.234768] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9029.336167] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9035.202442] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9043.911789] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9044.238366] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9048.473155] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9048.689962] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9053.606354] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9053.854846] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9058.738894] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9059.001422] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9064.930716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9064.945928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9064.968995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9065.343730] Lustre: server umount lustre-MDT0000 complete [ 9065.447660] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9069.568564] LustreError: 297148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788518996 with bad export cookie 4213754578593605175 [ 9069.609212] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9069.639845] LustreError: 297148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9070.298924] Lustre: server umount lustre-MDT0001 complete [ 9084.976977] Lustre: server umount lustre-OST2710 complete [ 9086.944393] Lustre: 294951:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788518997/real 1788518997] req@00000000d6abe5a0 x1875397909220224/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788519013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9086.989778] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9087.009738] Lustre: Skipped 2 previous similar messages [ 9090.199810] Lustre: server umount lustre-OST03e8 complete [ 9111.624813] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 9123.011740] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9161.059355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.217372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9161.287295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9161.352271] Lustre: lustre-MDT0000: new disk, initializing [ 9161.400872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9161.414149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9167.205465] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9180.517582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9180.619106] Lustre: 305441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9180.672651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9180.676101] Lustre: Skipped 1 previous similar message [ 9180.742336] Lustre: lustre-MDT0001: new disk, initializing [ 9180.835985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9180.882757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9180.889806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9185.903633] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9196.647628] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9201.787961] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9208.625420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9208.926669] Lustre: lustre-OST0000: new disk, initializing [ 9208.940511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9208.959679] Lustre: 307410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9209.094376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9215.288273] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9218.114909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9218.131533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9218.223375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9225.318527] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9230.875260] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9231.110665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9236.322162] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9236.593968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9239.009598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9239.014950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.023759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9239.035751] Lustre: Skipped 1 previous similar message [ 9239.056418] Lustre: Skipped 1 previous similar message [ 9244.139238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9244.150152] Lustre: Skipped 1 previous similar message [ 9244.794364] Lustre: server umount lustre-OST0000 complete [ 9249.253496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9249.278584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9249.286600] Lustre: Skipped 1 previous similar message [ 9254.370848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9254.380065] Lustre: Skipped 1 previous similar message [ 9255.672743] Lustre: server umount lustre-MDT0000 complete [ 9259.492388] LustreError: 305451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9259.519659] LustreError: 305451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9259.547778] LustreError: 305431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788519186 with bad export cookie 4213754578593606855 [ 9259.564683] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9259.567620] LustreError: 305431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9260.084578] Lustre: server umount lustre-MDT0001 complete [ 9269.358805] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:53:13 (1788519193) [ 9271.080958] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9273.484429] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:53:17 (1788519197) [ 9277.547343] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9305.489294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9305.811951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9305.856543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9306.068263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9310.347268] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9318.317811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.385524] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9318.415751] Lustre: 311267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9322.915232] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9343.931680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9344.220906] Lustre: Denying initial registration attempt from nid 192.168.203.151@tcp, specified as failover [ 9344.232235] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9344.241993] LustreError: 312720:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9344.247949] LustreError: 312720:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9344.254435] LustreError: 312720:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9344.323274] Lustre: server umount lustre-OST0000 complete [ 9344.327117] LustreError: 312720:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9351.137887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9351.166458] Lustre: Skipped 1 previous similar message [ 9351.174806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.185931] Lustre: Skipped 1 previous similar message [ 9356.893967] Lustre: server umount lustre-MDT0000 complete [ 9360.795743] LustreError: 311257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788519287 with bad export cookie 4213754578593607730 [ 9360.801969] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9360.809831] LustreError: 311257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9361.224413] Lustre: server umount lustre-MDT0001 complete [ 9370.466841] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9373.069849] Key type lgssc unregistered [ 9373.311892] LNet: 313838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9373.321745] LNetError: 313838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9373.337409] LNet: Removed LNI 192.168.203.151@tcp [ 9374.219210] Key type .llcrypt unregistered [ 9374.225122] Key type ._llcrypt unregistered [ 9385.381415] Key type ._llcrypt registered [ 9385.383145] Key type .llcrypt registered [ 9385.487620] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9386.383062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9386.439847] alg: No test for adler32 (adler32-zlib) [ 9387.496755] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 9387.711624] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9389.368227] Key type lgssc registered [ 9390.675852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9435.140584] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9443.913702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9443.957882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.281149] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9445.317606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9445.471266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9449.388349] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9457.398034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9457.495662] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9457.528237] Lustre: 316596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9457.744795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9462.511730] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9471.234689] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9476.957920] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9496.496279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9496.836674] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9497.012522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9504.069057] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9512.676234] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9519.587245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.590120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9519.604541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9519.628167] LustreError: Skipped 1 previous similar message [ 9524.707260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9524.717968] Lustre: Skipped 1 previous similar message [ 9529.312815] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9529.657717] Lustre: server umount lustre-OST0000 complete [ 9534.434628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9534.444409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9534.459245] Lustre: Skipped 1 previous similar message [ 9534.472258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9534.476952] Lustre: Skipped 1 previous similar message [ 9539.797710] Lustre: server umount lustre-MDT0000 complete [ 9544.522542] LustreError: 316587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788519471 with bad export cookie 18328135030956977895 [ 9544.523702] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9544.534344] LustreError: 316587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9544.834309] Lustre: server umount lustre-MDT0001 complete [ 9556.767935] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9559.793555] Key type lgssc unregistered [ 9560.120714] LNet: 320170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9560.156679] LNetError: 320170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9560.178079] LNet: Removed LNI 192.168.203.151@tcp [ 9561.134149] Key type .llcrypt unregistered [ 9561.136902] Key type ._llcrypt unregistered [ 9587.308192] Key type ._llcrypt registered [ 9587.311143] Key type .llcrypt registered [ 9587.403558] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9589.126727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9589.264489] alg: No test for adler32 (adler32-zlib) [ 9590.350866] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 9590.618710] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9592.353439] Key type lgssc registered [ 9593.734848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9602.291691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9602.325364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9603.950251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9609.285585] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9614.319774] LustreError: 321582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9614.357172] LustreError: 321582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9618.056309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9618.499572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9622.800045] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9630.866570] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9636.742480] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9646.330550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9646.838406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.938363] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9662.483711] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9673.699317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9673.726104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9673.736940] Lustre: Skipped 2 previous similar messages [ 9675.661789] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 9677.328811] Lustre: server umount lustre-MDT0000 complete [ 9678.822395] LustreError: 321577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.839080] LustreError: 321577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9680.749907] LustreError: 321579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9681.260656] LustreError: 321563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788519607 with bad export cookie 15081063377577740851 [ 9681.269272] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9681.278537] LustreError: 321563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9683.937901] LustreError: 322777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9683.938364] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.977691] Lustre: Skipped 2 previous similar messages [ 9683.989958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9686.947391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9686.955451] Lustre: Skipped 1 previous similar message [ 9687.931084] Lustre: server umount lustre-MDT0001 complete [ 9695.021619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9703.493369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9705.632532] LustreError: 325166:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9705.641049] LustreError: 325166:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000bc316cb0 x1875398641101440/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788519632 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9705.659662] LustreError: 325166:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9706.817311] LustreError: 323418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.841499] LustreError: 323418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9706.908991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9706.990970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9712.165738] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9716.137627] LustreError: 325179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9716.144932] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9716.167225] LustreError: 325179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9721.260377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9721.708196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9725.884495] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9726.973442] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9733.971800] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9736.254757] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9743.051449] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9744.662884] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9752.564865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9752.574924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.581202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9752.619055] Lustre: Skipped 1 previous similar message [ 9762.787572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9762.804678] Lustre: Skipped 4 previous similar messages [ 9764.321395] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9764.626431] Lustre: server umount lustre-OST0000 complete [ 9773.026831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9773.029233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9774.681816] Lustre: server umount lustre-MDT0000 complete [ 9778.153510] LustreError: 325183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9778.172281] LustreError: 325183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9779.556676] LustreError: 321562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788519706 with bad export cookie 15081063377577748628 [ 9779.568904] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9779.579922] LustreError: 321562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9779.906635] Lustre: server umount lustre-MDT0001 complete [ 9791.107680] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9794.183259] Key type lgssc unregistered [ 9794.486777] LNet: 327863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9794.494833] LNetError: 327863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9794.516169] LNet: Removed LNI 192.168.203.151@tcp [ 9795.451621] Key type .llcrypt unregistered [ 9795.457304] Key type ._llcrypt unregistered [ 9822.615703] Key type ._llcrypt registered [ 9822.617917] Key type .llcrypt registered [ 9822.717078] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9823.698267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9823.728923] alg: No test for adler32 (adler32-zlib) [ 9825.344547] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [ 9825.773960] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9827.480203] Key type lgssc registered [ 9828.916983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9837.482988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9837.505061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.175712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9844.210890] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9849.330806] LustreError: 329278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9849.355954] LustreError: 329278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9853.624162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9854.147996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9858.924662] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9867.575167] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9874.921429] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9879.013294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.040200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.047437] Lustre: Skipped 1 previous similar message [ 9884.131140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9884.137864] Lustre: Skipped 1 previous similar message [ 9889.249785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9889.262447] Lustre: Skipped 1 previous similar message [ 9892.320142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9892.599070] Lustre: server umount lustre-MDT0000 complete [ 9894.379415] LustreError: 329277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9894.401651] LustreError: 329277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9896.173320] LustreError: 329257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788519822 with bad export cookie 8832229171138189385 [ 9896.183304] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9896.189412] LustreError: 329257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9896.555832] Lustre: server umount lustre-MDT0001 complete [ 9913.710837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9914.159511] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9914.193192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9914.395890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.292188] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9928.383636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9928.558070] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9928.606747] Lustre: 331870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9928.722723] LustreError: 331881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.014867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9933.425132] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9941.074567] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9947.622860] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9959.156545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9959.432458] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9959.454579] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9959.466243] LustreError: 333755:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9959.479164] LustreError: 333755:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9959.491189] LustreError: 333755:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9959.568405] Lustre: server umount lustre-OST0000 complete [ 9959.576170] LustreError: 333755:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9971.760625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9972.002729] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9972.215431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9978.410578] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9980.561988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9986.771846] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9995.632410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9995.723948] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9995.884818] Lustre: lustre-OST0001: new disk, initializing [ 9995.889118] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9995.897924] Lustre: 335297:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9996.004725] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10004.266773] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10005.148724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10005.159564] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10005.275613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10012.928313] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10020.834022] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10020.847554] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10020.851161] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10020.865803] Lustre: Skipped 2 previous similar messages [10020.895412] Lustre: Skipped 1 previous similar message [10025.955385] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10025.966353] Lustre: Skipped 1 previous similar message [10030.051326] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10030.212758] Lustre: server umount lustre-OST0001 complete [10031.078573] LustreError: 334222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10031.111334] LustreError: 334222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10036.205335] LustreError: 334221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.206652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.215481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10036.231279] LustreError: 334221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10036.251460] Lustre: Skipped 1 previous similar message [10040.795388] Lustre: server umount lustre-OST0000 complete [10046.445811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10051.039718] Lustre: server umount lustre-MDT0000 complete [10051.552990] LustreError: 331882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.576136] LustreError: 331882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10054.623983] LustreError: 331863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788519981 with bad export cookie 8832229171138189938 [10054.641073] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10054.929691] Lustre: server umount lustre-MDT0001 complete [10066.775096] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [10070.025796] Key type lgssc unregistered [10070.295879] LNet: 337256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10070.315370] LNetError: 337256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10070.336814] LNet: Removed LNI 192.168.203.151@tcp [10071.380222] Key type .llcrypt unregistered [10071.382567] Key type ._llcrypt unregistered [10142.873439] Key type ._llcrypt registered [10142.877523] Key type .llcrypt registered [10143.001338] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [10158.435190] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10159.550578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10159.590179] alg: No test for adler32 (adler32-zlib) [10160.602044] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [10160.852169] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [10162.616192] Key type lgssc registered [10163.676561] Lustre: Echo OBD driver; http://www.lustre.org/ [10198.939984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10198.958284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10200.223252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10200.272751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10200.353843] Lustre: lustre-MDT0000: new disk, initializing [10200.470061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10200.492085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10205.138806] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10215.872830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.012942] Lustre: 342189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10216.135613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10216.143996] Lustre: Skipped 1 previous similar message [10216.273949] Lustre: lustre-MDT0001: new disk, initializing [10216.394467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10216.450572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10216.462264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10221.275434] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10230.723988] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10236.238313] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10243.339195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10243.580604] Lustre: lustre-OST0000: new disk, initializing [10243.585252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10243.593358] Lustre: 344160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10243.681238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10249.310421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10249.318395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10249.402610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10249.754503] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10257.904364] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10262.363542] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10262.603126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10266.761603] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10266.968400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10270.179562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10270.187946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.195217] LustreError: Skipped 1 previous similar message [10270.199040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10270.214366] Lustre: Skipped 1 previous similar message [10274.894238] Lustre: server umount lustre-OST0000 complete [10280.417743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10280.424997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10280.434338] Lustre: Skipped 1 previous similar message [10280.446377] Lustre: Skipped 2 previous similar messages [10285.537547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10285.544245] Lustre: Skipped 1 previous similar message [10289.638205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10289.649431] Lustre: Skipped 1 previous similar message [10292.192124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10294.401936] Lustre: server umount lustre-MDT0000 complete [10298.180247] LustreError: 342183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788520224 with bad export cookie 286833528273992863 [10298.193653] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10298.198375] LustreError: 342183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10298.602876] Lustre: server umount lustre-MDT0001 complete [10306.950742] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:10:31 (1788520231) [10315.300940] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:10:39 (1788520239) [10324.903195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.378749] LustreError: 346917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.399562] LustreError: 346917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10325.453444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10330.613657] LustreError: 346918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.614947] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10334.691571] LustreError: 346917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10339.513842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10339.884068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10345.262934] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10352.943840] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10358.293991] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10364.705632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10365.005105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10372.100481] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10378.973862] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10386.837351] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10388.370668] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10390.058433] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10392.135571] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10394.288182] Lustre: Failing over lustre-MDT0000 [10394.697831] Lustre: server umount lustre-MDT0000 complete [10396.135737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10396.139885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10396.149031] LustreError: 348987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10396.191058] Lustre: Skipped 1 previous similar message [10401.251174] LustreError: 346918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.290561] LustreError: 346918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10411.496071] LustreError: 346917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10411.517376] LustreError: 346917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10412.512137] Lustre: 340414:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788520322/real 1788520322] req@00000000044aba3a x1875399238816384/t0(0) o400->MGC192.168.203.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788520338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10412.551506] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10414.730383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.753503] LustreError: 340411:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008d921029 x1875399238822016/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10423.305745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10423.404172] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10427.767808] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10428.411902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10428.437016] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10428.496971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10428.764522] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10441.162951] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10443.077720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10447.270707] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10449.328994] Lustre: Failing over lustre-MDT0000 [10451.628881] Lustre: server umount lustre-MDT0000 complete [10453.397651] LustreError: 346913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10453.415086] LustreError: 346913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10454.001675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10454.004181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10454.023162] Lustre: Skipped 2 previous similar messages [10470.304961] Lustre: 340415:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788520380/real 1788520380] req@0000000041043339 x1875399238840064/t0(0) o400->MGC192.168.203.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788520396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10470.347775] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10471.088920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.841650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10479.895636] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10479.982115] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10485.247708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10485.262712] Lustre: Skipped 2 previous similar messages [10485.312175] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [10485.383174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10485.840664] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10498.018627] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10500.570164] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10510.826163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.837719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10510.858160] Lustre: Skipped 2 previous similar messages [10510.896030] Lustre: Skipped 2 previous similar messages [10514.638288] Lustre: server umount lustre-MDT0000 complete [10515.939506] LustreError: 346917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10515.962638] LustreError: 346917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [10518.209276] LustreError: 346898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788520444 with bad export cookie 286833528273995572 [10518.216315] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10518.225644] LustreError: 346898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10518.603709] Lustre: server umount lustre-MDT0001 complete [10533.031057] Lustre: server umount lustre-OST0000 complete [10548.986641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10549.470211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10553.711707] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10562.587771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10568.393147] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10575.636376] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10580.959758] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10587.760986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10594.294686] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10594.300187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10601.585257] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10619.884485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10619.892957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10619.920651] Lustre: Skipped 1 previous similar message [10619.929597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10624.739176] Lustre: server umount lustre-OST0000 complete [10636.454894] Lustre: server umount lustre-MDT0000 complete [10638.306245] LustreError: 353268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.362972] LustreError: 353268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10641.513564] LustreError: 353249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788520568 with bad export cookie 286833528273996321 [10641.544758] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.075785] Lustre: server umount lustre-MDT0001 complete [10654.815871] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [10658.219285] Key type lgssc unregistered [10658.619891] LNet: 356822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10658.630727] LNetError: 356822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10658.662073] LNet: Removed LNI 192.168.203.151@tcp [10659.533606] Key type .llcrypt unregistered [10659.536728] Key type ._llcrypt unregistered [10688.232643] Key type ._llcrypt registered [10688.236377] Key type .llcrypt registered [10688.361673] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10689.160894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10689.232033] alg: No test for adler32 (adler32-zlib) [10690.387451] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [10690.635686] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [10692.426569] Key type lgssc registered [10694.254302] Lustre: Echo OBD driver; http://www.lustre.org/ [10703.202795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10703.223980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10704.920869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10709.933412] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10715.110555] LustreError: 358335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.133887] LustreError: 358335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10719.351698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10719.778869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10724.376103] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10732.257607] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10738.921272] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10746.070299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10746.508346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10749.566374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10752.281431] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10760.371547] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10775.526816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.531737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10775.556017] Lustre: Skipped 2 previous similar messages [10775.571518] Lustre: Skipped 2 previous similar messages [10777.488671] Lustre: server umount lustre-MDT0000 complete [10780.643182] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.673853] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10780.903780] LustreError: 358316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788520707 with bad export cookie 10705398066964822939 [10780.911647] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10780.924065] LustreError: 358316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10781.272969] Lustre: server umount lustre-MDT0001 complete [10795.674157] Lustre: server umount lustre-OST0000 complete [10807.515357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.967532] LustreError: 362068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.067226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10812.467686] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10813.421180] LustreError: 362069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.455518] LustreError: 362069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10817.511421] LustreError: 362068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.490778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10821.950292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10826.723960] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10834.147478] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10839.048702] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10845.716601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10846.051437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10852.278973] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10853.265412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10859.710957] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10873.313510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10873.330353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10873.359458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10878.666922] Lustre: server umount lustre-MDT0000 complete [10878.954722] LustreError: 362068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10881.948569] LustreError: 362049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788520808 with bad export cookie 10705398066964824052 [10881.955186] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10881.964694] LustreError: 362049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10882.304216] Lustre: server umount lustre-MDT0001 complete [10896.133598] Lustre: server umount lustre-OST0000 complete [10913.108747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.537247] LustreError: 365979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.559316] LustreError: 365979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10913.605383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10918.388589] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10926.748547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.011173] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10927.022515] Lustre: Skipped 2 previous similar messages [10932.143198] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10940.636339] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10947.778085] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10955.506325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10956.017824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10956.029693] Lustre: Skipped 1 previous similar message [10959.100763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10962.241888] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10970.056743] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10977.761078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10977.768153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10977.781427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10977.803652] Lustre: Skipped 3 previous similar messages [10981.895832] Lustre: server umount lustre-OST0000 complete [10988.004230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.009198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10988.014294] Lustre: Skipped 1 previous similar message [10988.027212] Lustre: Skipped 2 previous similar messages [10998.243431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10998.247280] Lustre: Skipped 3 previous similar messages [11000.288177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11000.635571] Lustre: server umount lustre-MDT0000 complete [11003.362910] LustreError: 365979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11003.382064] LustreError: 365979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11004.392968] LustreError: 365960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788520930 with bad export cookie 10705398066964825074 [11004.404346] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11004.410362] LustreError: 365960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11004.775776] Lustre: server umount lustre-MDT0001 complete [11017.297796] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11020.113817] Key type lgssc unregistered [11020.417515] LNet: 369513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11020.425457] LNetError: 369513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11020.441776] LNet: Removed LNI 192.168.203.151@tcp [11021.337300] Key type .llcrypt unregistered [11021.340836] Key type ._llcrypt unregistered [11048.137253] Key type ._llcrypt registered [11048.139131] Key type .llcrypt registered [11048.240570] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11048.925594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11049.038736] alg: No test for adler32 (adler32-zlib) [11050.240855] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [11050.551626] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11052.304305] Key type lgssc registered [11053.632774] Lustre: Echo OBD driver; http://www.lustre.org/ [11062.307260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11062.332328] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11062.342555] LustreError: 370950:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11062.368147] LustreError: 370950:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11062.378346] LustreError: 370950:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11062.384555] LustreError: 370950:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11062.391773] LustreError: 370950:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11070.959828] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11070.968627] LustreError: 371348:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11070.985926] LustreError: 371348:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11070.991639] LustreError: 371348:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11070.997883] LustreError: 371348:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11071.004393] LustreError: 371348:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11091.427669] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11094.354918] Key type lgssc unregistered [11094.675609] LNet: 372352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11094.680025] LNetError: 372352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11094.692670] LNet: Removed LNI 192.168.203.151@tcp [11095.696546] Key type .llcrypt unregistered [11095.717841] Key type ._llcrypt unregistered [11114.320387] Key type ._llcrypt registered [11114.323803] Key type .llcrypt registered [11114.446535] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11130.256474] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11131.201084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11131.222051] alg: No test for adler32 (adler32-zlib) [11132.505987] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [11132.769037] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11134.481173] Key type lgssc registered [11135.793065] Lustre: Echo OBD driver; http://www.lustre.org/ [11177.364097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11177.390721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11178.663334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11178.698281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11178.781210] Lustre: lustre-MDT0000: new disk, initializing [11178.846043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11178.871634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11183.491162] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11194.647404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.775091] Lustre: 375508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11194.856497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11194.869190] Lustre: Skipped 1 previous similar message [11194.966123] Lustre: lustre-MDT0001: new disk, initializing [11195.100400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11195.152820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11195.168170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11199.815458] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11210.343530] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11216.798652] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11224.951029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11225.375996] Lustre: lustre-OST0000: new disk, initializing [11225.381528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11225.391513] Lustre: 377477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11225.524818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11232.677835] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11235.382803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11235.393072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11235.478490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11243.246625] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11248.522862] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11248.724211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11253.546515] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11253.849247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11256.289525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11256.301648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11256.310869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11256.315091] Lustre: Skipped 1 previous similar message [11261.421291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11261.428253] Lustre: Skipped 2 previous similar messages [11261.831374] Lustre: server umount lustre-OST0000 complete [11266.535361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11266.547088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11266.553311] Lustre: Skipped 1 previous similar message [11266.569970] Lustre: Skipped 1 previous similar message [11271.858343] Lustre: server umount lustre-MDT0000 complete [11275.817818] LustreError: 375500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788521202 with bad export cookie 12892656660072183025 [11275.835442] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11275.841570] LustreError: 375500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11276.199692] Lustre: server umount lustre-MDT0001 complete [11285.568215] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11287.257732] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:26:51 (1788521211) [11294.701208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11295.161119] LustreError: 379936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11295.185424] LustreError: 379936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11295.249379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11299.983530] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11300.328479] LustreError: 379937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.458742] LustreError: 379936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.315946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.597253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11312.742729] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11321.789701] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11327.962957] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11335.159548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11335.595857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11343.117676] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11351.881080] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11359.250479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11359.416267] Lustre: lustre-OST0001: new disk, initializing [11359.418860] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11359.432804] Lustre: 382769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11359.593274] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11366.209754] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11369.055588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11369.066200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11369.214935] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11376.603860] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11384.201629] Lustre: lustre-OST0001: Not available for connect from 192.168.203.51@tcp (stopping) [11384.803968] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.819877] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11384.854185] LustreError: Skipped 1 previous similar message [11387.546835] Lustre: server umount lustre-OST0001 complete [11389.305759] LustreError: 381758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11395.052965] LustreError: 381757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11395.100811] LustreError: 381757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11405.289210] LustreError: 381755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.316633] LustreError: 381755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11406.309578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11406.326080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11406.335588] Lustre: Skipped 1 previous similar message [11406.343246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11406.350166] Lustre: Skipped 2 previous similar messages [11410.417917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11412.616423] Lustre: server umount lustre-OST0000 complete [11420.661714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11420.664466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11420.685408] Lustre: Skipped 1 previous similar message [11420.707518] Lustre: Skipped 3 previous similar messages [11431.392218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11433.639447] Lustre: server umount lustre-MDT0000 complete [11436.013205] LustreError: 379931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.034108] LustreError: 379931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11437.177664] LustreError: 379917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788521363 with bad export cookie 12892656660072183907 [11437.179021] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11437.190096] LustreError: 379917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11443.569431] Lustre: server umount lustre-MDT0001 complete [11454.031024] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11457.078364] Key type lgssc unregistered [11457.573721] LNet: 384757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11457.583762] LNetError: 384757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11457.606927] LNet: Removed LNI 192.168.203.151@tcp [11458.727932] Key type .llcrypt unregistered [11458.731778] Key type ._llcrypt unregistered [11506.907979] Key type ._llcrypt registered [11506.909536] Key type .llcrypt registered [11506.999811] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11507.912106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11508.022507] alg: No test for adler32 (adler32-zlib) [11509.115969] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [11509.469953] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11511.136327] Key type lgssc registered [11512.257538] Lustre: Echo OBD driver; http://www.lustre.org/ [11519.796438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11519.813760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11521.082526] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11521.098303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11521.298975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11525.692653] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11533.955868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11534.093447] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11534.137199] Lustre: 387553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11534.458813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11539.190612] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11547.612307] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11553.399366] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11558.370405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.374810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11558.386664] Lustre: Skipped 1 previous similar message [11558.408065] Lustre: Skipped 1 previous similar message [11561.728684] Lustre: server umount lustre-MDT0000 complete [11565.446334] LustreError: 389148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788521491 with bad export cookie 15211060101709253035 [11565.447198] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11565.463695] LustreError: 389148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11565.851186] Lustre: server umount lustre-MDT0001 complete [11575.823574] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:31:40 (1788521500) [11582.180121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11582.627612] LustreError: 390067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11582.654579] LustreError: 390067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11582.709787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11587.113723] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11588.065125] LustreError: 390068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.190863] LustreError: 390067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11595.545111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11595.955360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11600.251858] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11607.436551] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11614.262447] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11621.573692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11621.731527] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11621.940775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11628.572160] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11636.566767] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11644.782783] Lustre: Permanently deactivating lustre-OST0000 [11644.798677] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11654.170723] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11657.190161] LustreError: 393101:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11657.207248] LustreError: 393101:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11657.220676] LustreError: 393101:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11661.997517] LustreError: 393150:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11662.010083] LustreError: 393150:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11666.978129] Lustre: server umount lustre-OST0000 complete [11669.853932] LustreError: 393400:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11669.867314] LustreError: 393400:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11673.065597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11673.069221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11673.097036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11673.097302] Lustre: Skipped 1 previous similar message [11678.178819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11678.187600] Lustre: Skipped 2 previous similar messages [11683.299140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11683.309260] Lustre: Skipped 1 previous similar message [11686.368770] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11686.546609] Lustre: server umount lustre-MDT0000 complete [11688.424220] LustreError: 390062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11688.474851] LustreError: 390062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11690.120665] LustreError: 390047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788521616 with bad export cookie 15211060101709253616 [11690.133333] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11690.136469] LustreError: 390047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11700.024153] Lustre: server umount lustre-MDT0001 complete [11710.525183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11714.922746] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11718.271154] LustreError: 394559:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11724.839902] Lustre: 394898:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.151@tcp [11727.288532] Lustre: server umount MGS complete [11738.310135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11738.898850] LustreError: 395620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11738.924104] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11738.930986] Lustre: Skipped 1 previous similar message [11738.988474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11743.911237] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11749.351367] LustreError: 395621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11749.393761] LustreError: 395621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11752.912509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11753.478233] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11759.133721] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11768.164328] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11775.115196] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11777.593577] Lustre: Permanently reactivating lustre-OST0000 [11777.615276] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11777.622914] Lustre: Skipped 2 previous similar messages [11805.152574] LustreError: 397361:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11815.147701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11815.828395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11815.852771] Lustre: Skipped 1 previous similar message [11823.692629] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11831.412393] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11840.485112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11840.504037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11840.516855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11840.524656] Lustre: Skipped 1 previous similar message [11840.550338] LustreError: Skipped 1 previous similar message [11843.341980] Lustre: server umount lustre-OST0000 complete [11850.724443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11850.733031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11850.737609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11850.751349] Lustre: Skipped 1 previous similar message [11850.759392] Lustre: Skipped 2 previous similar messages [11853.666542] Lustre: server umount lustre-MDT0000 complete [11855.848676] LustreError: 395620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11855.884876] LustreError: 395620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11857.941542] LustreError: 395602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788521784 with bad export cookie 15211060101709254666 [11857.956251] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11858.430628] Lustre: server umount lustre-MDT0001 complete [11869.898470] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11873.155944] Key type lgssc unregistered [11873.509342] LNet: 399987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11873.521221] LNetError: 399987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11873.541934] LNet: Removed LNI 192.168.203.151@tcp [11874.605994] Key type .llcrypt unregistered [11874.608184] Key type ._llcrypt unregistered [11893.964818] Key type ._llcrypt registered [11893.967367] Key type .llcrypt registered [11894.156231] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11910.357475] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11911.541613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11911.570471] alg: No test for adler32 (adler32-zlib) [11912.801574] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [11913.129541] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11914.784390] Key type lgssc registered [11916.056409] Lustre: Echo OBD driver; http://www.lustre.org/ [11955.318532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11955.337900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11956.620452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11956.661443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11956.798944] Lustre: lustre-MDT0000: new disk, initializing [11956.882771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11956.930692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11961.846712] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11975.006088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11975.181742] Lustre: 403142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11975.258869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11975.265539] Lustre: Skipped 1 previous similar message [11975.399819] Lustre: lustre-MDT0001: new disk, initializing [11975.515407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11975.578328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11975.597620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11981.883934] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11991.901622] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11997.559039] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12004.119151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12004.432145] Lustre: lustre-OST0000: new disk, initializing [12004.445564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12004.462499] Lustre: 405111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12004.619908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12011.536498] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12011.598868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12011.612428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12011.750354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12022.308561] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12027.505983] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12027.807820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12032.769569] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12032.955822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12037.607839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12037.622423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12037.631247] LustreError: Skipped 1 previous similar message [12037.650570] Lustre: Skipped 1 previous similar message [12037.654920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12037.670983] Lustre: Skipped 1 previous similar message [12040.952238] Lustre: server umount lustre-OST0000 complete [12047.329702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12047.344677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12047.358186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12052.962667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12052.972406] Lustre: Skipped 2 previous similar messages [12058.082305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12058.090058] Lustre: Skipped 1 previous similar message [12059.104163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12059.466691] Lustre: server umount lustre-MDT0000 complete [12063.443891] LustreError: 403134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788521989 with bad export cookie 963010543995854833 [12063.444960] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12063.452302] LustreError: 403134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12063.974642] Lustre: server umount lustre-MDT0001 complete [12075.034793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12077.257894] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:40:01 (1788522001) [12085.656421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12086.165038] LustreError: 407572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12086.196140] LustreError: 407572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12086.264796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12090.759546] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12091.368398] LustreError: 407573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12096.483881] LustreError: 407572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12100.190712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12100.801831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12105.618418] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12113.981801] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12121.863400] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12128.984629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12129.315252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12136.398150] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12144.588490] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12147.010866] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12150.840777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12157.922392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12157.935344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.955624] Lustre: Skipped 1 previous similar message [12157.964661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12160.997162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12161.008892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12162.214743] Lustre: server umount lustre-OST0000 complete [12166.113972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12166.143601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12166.157027] Lustre: Skipped 3 previous similar messages [12172.346917] Lustre: server umount lustre-MDT0000 complete [12176.253138] LustreError: 407552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788522102 with bad export cookie 963010543995855715 [12176.272037] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12176.276066] LustreError: 407552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12176.777238] Lustre: server umount lustre-MDT0001 complete [12188.515786] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12191.447478] Key type lgssc unregistered [12191.752694] LNet: 411296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12191.763742] LNetError: 411296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12191.780763] LNet: Removed LNI 192.168.203.151@tcp [12192.741459] Key type .llcrypt unregistered [12192.749360] Key type ._llcrypt unregistered [12218.824712] Key type ._llcrypt registered [12218.827501] Key type .llcrypt registered [12218.923149] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12219.175404] Key type .llcrypt unregistered [12219.179947] Key type ._llcrypt unregistered [12237.127917] Key type ._llcrypt registered [12237.134559] Key type .llcrypt registered [12237.238871] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12238.219176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12238.366685] alg: No test for adler32 (adler32-zlib) [12239.423792] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [12239.756164] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12241.548679] Key type lgssc registered [12242.961787] Lustre: Echo OBD driver; http://www.lustre.org/ [12251.572234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12251.605475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.064027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12258.093578] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12263.406218] LustreError: 413503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12263.445487] LustreError: 413503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12266.748793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12267.289408] LustreError: 413504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12267.346434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12272.609685] LustreError: 413503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12272.625756] LustreError: 413503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12273.814319] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12277.739922] LustreError: 413504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.752733] LustreError: 413504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12277.753714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12280.310970] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12282.861699] LustreError: 413500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.899038] LustreError: 413500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12287.112238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12287.456098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12292.549322] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12302.818854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12302.822824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12302.833534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12302.845471] Lustre: Skipped 1 previous similar message [12302.865711] Lustre: Skipped 1 previous similar message [12307.942895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12307.949144] Lustre: Skipped 1 previous similar message [12313.064284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12313.074610] Lustre: Skipped 1 previous similar message [12316.128195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12316.532989] Lustre: server umount lustre-OST0000 complete [12323.300616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.318596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12327.052132] Lustre: server umount lustre-MDT0000 complete [12328.422867] LustreError: 413499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12328.444886] LustreError: 413499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12330.831860] LustreError: 413485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788522257 with bad export cookie 10199904048018764960 [12330.839974] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12330.848497] LustreError: 413485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12331.115906] Lustre: server umount lustre-MDT0001 complete [12341.838260] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12344.770734] Key type lgssc unregistered [12345.121157] LNet: 416651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12345.128785] LNetError: 416651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12345.147610] LNet: Removed LNI 192.168.203.151@tcp [12346.088660] Key type .llcrypt unregistered [12346.090220] Key type ._llcrypt unregistered [12370.633399] Key type ._llcrypt registered [12370.636512] Key type .llcrypt registered [12370.759573] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12372.240951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12372.294962] alg: No test for adler32 (adler32-zlib) [12373.434615] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [12373.716144] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12375.368142] Key type lgssc registered [12376.620730] Lustre: Echo OBD driver; http://www.lustre.org/ [12384.216733] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12384.250906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12399.840684] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12404.961057] LustreError: 418040:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.151@tcp: failed processing log, type 4: rc = -110 [12431.648235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12439.458940] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12448.108851] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12456.223687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12456.706666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12456.801983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12462.054023] LustreError: 418065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12462.073759] LustreError: 418065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12462.123818] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12466.157936] LustreError: 419079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12466.175089] LustreError: 419079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12471.267947] LustreError: 419077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12471.298167] LustreError: 419077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12472.935501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12473.510707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12479.774124] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12490.692708] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12498.641753] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12508.130955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12508.142949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12508.197238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12509.156926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12509.183269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12509.212927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12514.519746] Lustre: server umount lustre-OST0000 complete [12519.394655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12519.411356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12519.434890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12519.440813] Lustre: Skipped 1 previous similar message [12521.059608] Lustre: server umount lustre-MDT0000 complete [12525.029611] LustreError: 419083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12525.072810] LustreError: 419083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12525.547081] LustreError: 418048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788522452 with bad export cookie 256690486421796237 [12525.554157] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12525.557212] LustreError: 418048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12526.070074] Lustre: server umount lustre-MDT0001 complete [12538.449641] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12541.516276] Key type lgssc unregistered [12541.924824] LNet: 421609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12541.931240] LNetError: 421609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12541.945843] LNet: Removed LNI 192.168.203.151@tcp [12542.825116] Key type .llcrypt unregistered [12542.832273] Key type ._llcrypt unregistered [12569.572904] Key type ._llcrypt registered [12569.576704] Key type .llcrypt registered [12569.706382] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12570.907652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12571.031730] alg: No test for adler32 (adler32-zlib) [12572.224874] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [12572.515202] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12574.194603] Key type lgssc registered [12575.567371] Lustre: Echo OBD driver; http://www.lustre.org/ [12585.113454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12585.142764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12586.796835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12591.737689] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12597.235942] LustreError: 423026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12597.282153] LustreError: 423026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12601.209197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12601.701442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12605.860577] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12615.033773] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12622.046620] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12631.168337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12631.641653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12636.653313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12638.382498] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12646.654857] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12653.031690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.043692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12653.044561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12653.052938] Lustre: Skipped 1 previous similar message [12654.476429] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [12654.484269] Lustre: Skipped 2 previous similar messages [12657.728756] Lustre: server umount lustre-MDT0000 complete [12658.150642] LustreError: 423022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12658.166144] LustreError: 423022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12659.561206] LustreError: 425603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12659.996811] LustreError: 425602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.51@tcp arrived at 1788522586 with bad export cookie 12120077649241901104 [12661.943484] LustreError: 423006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788522588 with bad export cookie 12120077649241900208 [12661.955259] LustreError: 423006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12661.962032] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12663.265992] LustreError: 423026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12663.270454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12663.271076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12663.288738] LustreError: 423026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12663.331512] Lustre: Skipped 1 previous similar message [12663.340607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12666.277300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12668.262817] Lustre: server umount lustre-OST0000 complete [12671.459613] LustreError: 423022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12671.478562] LustreError: 423022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12676.343676] Lustre: server umount lustre-MDT0001 complete [12688.244913] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12690.894486] Key type lgssc unregistered [12691.176510] LNet: 426709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12691.184030] LNetError: 426709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12691.207112] LNet: Removed LNI 192.168.203.151@tcp [12692.043533] Key type .llcrypt unregistered [12692.045434] Key type ._llcrypt unregistered [12718.576732] Key type ._llcrypt registered [12718.579621] Key type .llcrypt registered [12718.671484] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12719.990981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12720.070777] alg: No test for adler32 (adler32-zlib) [12721.093871] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [12721.308847] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12723.065134] Key type lgssc registered [12724.672258] Lustre: Echo OBD driver; http://www.lustre.org/ [12735.436458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12735.493250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12737.375185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12744.032452] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12746.734679] LustreError: 428124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12746.760343] LustreError: 428124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12751.844458] LustreError: 428125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12754.260616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12754.663937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12759.430317] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12767.844528] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12773.614890] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12779.906473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12780.221393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12782.346863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12786.897894] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12795.600671] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12800.995449] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12801.009249] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12801.043665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12803.435807] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [12805.093246] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12805.094056] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12806.758765] Lustre: server umount lustre-MDT0001 complete [12808.557461] LustreError: 428119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12813.281484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12813.288305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12813.307465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12813.319125] Lustre: Skipped 1 previous similar message [12815.338362] LustreError: 428125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12815.364656] LustreError: 428125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12819.076782] Lustre: server umount lustre-OST0000 complete [12820.453096] LustreError: 428124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12823.184912] Lustre: server umount lustre-MDT0000 complete [12836.828145] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12839.803834] Key type lgssc unregistered [12840.185655] LNet: 431809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12840.196769] LNetError: 431809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12841.257327] LNet: Removed LNI 192.168.203.151@tcp [12842.232891] Key type .llcrypt unregistered [12842.235386] Key type ._llcrypt unregistered [12866.893253] Key type ._llcrypt registered [12866.896154] Key type .llcrypt registered [12867.003380] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [12881.855406] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12882.854235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12882.999908] alg: No test for adler32 (adler32-zlib) [12884.081078] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [12884.362233] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12886.080766] Key type lgssc registered [12887.289327] Lustre: Echo OBD driver; http://www.lustre.org/ [12924.243486] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12942.828411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12942.860116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12944.199718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12944.237943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12944.336789] Lustre: lustre-MDT0000: new disk, initializing [12944.472749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12944.492574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12949.293604] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12965.537663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12965.890395] Lustre: lustre-OST0000: new disk, initializing [12965.894211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12965.897984] Lustre: Skipped 1 previous similar message [12965.905820] Lustre: 436839:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12965.989408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12972.215759] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12975.707457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12975.716553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12975.758690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12980.851680] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12995.001305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12995.096489] Lustre: 435821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12995.148926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12995.214379] Lustre: lustre-MDT0001: new disk, initializing [12995.314022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12995.358488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12995.374196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12999.174619] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13008.306707] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13008.516960] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13012.563175] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13012.795677] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13017.058298] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13017.064153] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13017.074518] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13021.561933] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [13021.683635] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13026.664865] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [13026.674535] Lustre: Skipped 2 previous similar messages [13031.392945] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13031.775065] Lustre: server umount lustre-MDT0001 complete [13037.028994] LustreError: 437683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13037.065390] LustreError: 437683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13042.150137] LustreError: 435832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13042.186347] LustreError: 435832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13047.277807] LustreError: 435828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13047.306229] LustreError: 435828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13052.396574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13052.397793] LustreError: 435833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13052.415678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13052.466659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13052.479467] Lustre: Skipped 2 previous similar messages [13056.234779] Lustre: server umount lustre-OST0000 complete [13057.505790] LustreError: 435832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13066.742987] Lustre: server umount lustre-MDT0000 complete [13081.672854] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13084.172722] Key type lgssc unregistered [13084.411318] LNet: 440586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13084.417616] LNetError: 440586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13084.436123] LNet: Removed LNI 192.168.203.151@tcp [13085.282585] Key type .llcrypt unregistered [13085.286626] Key type ._llcrypt unregistered [13123.343994] Key type ._llcrypt registered [13123.348320] Key type .llcrypt registered [13123.454284] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13123.692501] Key type .llcrypt unregistered [13123.694203] Key type ._llcrypt unregistered [13140.423123] Key type ._llcrypt registered [13140.427257] Key type .llcrypt registered [13140.593907] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [13141.492111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13141.697246] alg: No test for adler32 (adler32-zlib) [13142.732879] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [13143.028172] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [13144.752201] Key type lgssc registered [13145.908838] Lustre: Echo OBD driver; http://www.lustre.org/ [13172.240716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13172.301344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13173.670179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13173.707698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13173.866259] Lustre: lustre-MDT0000: new disk, initializing [13173.962748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13173.991818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13178.331546] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13189.340179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13189.466836] Lustre: 443965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13189.573758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13189.585279] Lustre: Skipped 1 previous similar message [13189.712801] Lustre: lustre-MDT0001: new disk, initializing [13189.856052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13189.926454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13189.938756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13194.542243] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13204.705870] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13211.024749] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13218.130748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13218.529913] Lustre: lustre-OST0000: new disk, initializing [13218.534636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13218.547850] Lustre: 445931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13218.655181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13224.526363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13224.535337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13224.667730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13224.784593] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13233.542838] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13240.292652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13240.298335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13240.316515] Lustre: Skipped 2 previous similar messages [13240.330326] Lustre: Skipped 2 previous similar messages [13245.412210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13245.426563] Lustre: Skipped 2 previous similar messages [13245.719815] Lustre: server umount lustre-MDT0000 complete [13249.388860] LustreError: 443955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523175 with bad export cookie 8125037790437334676 [13249.394667] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13249.400656] LustreError: 443955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13249.740826] Lustre: server umount lustre-MDT0001 complete [13262.764877] Lustre: server umount lustre-OST0000 complete [13271.506720] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:59:56 (1788523196) [13278.219134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13278.895065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13282.625261] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13284.329147] LustreError: 447956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13284.340626] LustreError: 447956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13289.443458] LustreError: 447957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13289.488209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13289.910309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13293.574806] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13299.267301] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13304.716233] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13323.388556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13323.610971] Lustre: Found index 0 for lustre-OST0000, updating log [13323.742454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13324.780923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13329.439474] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13335.985758] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13340.129798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13340.136306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13340.150731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13346.358535] Lustre: server umount lustre-OST0000 complete [13361.636191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13361.661610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13361.678102] Lustre: Skipped 1 previous similar message [13361.692248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13361.699678] Lustre: Skipped 3 previous similar messages [13366.754995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13366.759365] Lustre: Skipped 2 previous similar messages [13367.465095] Lustre: server umount lustre-MDT0000 complete [13367.779098] LustreError: 447956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13367.802791] LustreError: 447956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13370.428962] LustreError: 447936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788523296 with bad export cookie 8125037790437336069 [13370.432299] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13370.441163] LustreError: 447936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13370.749887] Lustre: server umount lustre-MDT0001 complete [13378.281262] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:01:42 (1788523302) [13393.952453] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp [13993.427127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13994.063077] LustreError: 454374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13994.153213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13998.813920] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13999.585182] LustreError: 454375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14003.693602] LustreError: 454374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14006.728462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14007.222951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14011.179967] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14017.000477] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14021.727160] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14041.014431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14041.346253] Lustre: Found index 0 for lustre-OST0000, updating log [14041.671279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14042.692028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [14046.246113] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp [14046.924255] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14052.943366] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14055.851058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14055.864574] Lustre: Skipped 1 previous similar message [14055.876082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14069.217196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14069.353995] Lustre: server umount lustre-OST0000 complete [14073.828183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14073.834088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14073.851754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14074.918078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14075.938823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14075.955026] Lustre: Skipped 3 previous similar messages [14079.524547] Lustre: server umount lustre-MDT0000 complete [14080.039605] LustreError: 454374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14080.064312] LustreError: 454374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14083.077661] LustreError: 454356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524009 with bad export cookie 8125037790437337147 [14083.081043] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14083.090407] LustreError: 454356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14083.128325] LustreError: 442493:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c563ef9d x1875402365731968/t0(0) o8->lustre-OST0000-osc-MDT0001@111.222.173.27@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14083.445959] Lustre: server umount lustre-MDT0001 complete [14105.637504] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp [14116.812915] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [14130.260378] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14169.382379] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:14:53 (1788524093) [14187.783723] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [14190.999239] Key type lgssc unregistered [14191.530738] LNet: 460992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14191.545538] LNetError: 460992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14305.260421] LNet: Removed LNI 192.168.203.151@tcp [14306.205415] Key type .llcrypt unregistered [14306.208235] Key type ._llcrypt unregistered [14322.141574] Key type ._llcrypt registered [14322.146421] Key type .llcrypt registered [14322.252489] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14323.257241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14323.274918] alg: No test for adler32 (adler32-zlib) [14324.381371] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [14324.619757] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [14326.312491] Key type lgssc registered [14327.513178] Lustre: Echo OBD driver; http://www.lustre.org/ [14331.812608] LNet: Added LNI 192.168.203.151@tcp100 [8/256/0/180] [14332.620880] LNet: Added LNI 192.168.203.151@tcp101 [8/256/0/180] [14333.432889] LNet: Added LNI 192.168.203.151@tcp102 [8/256/0/180] [14334.328268] LNet: Added LNI 192.168.203.151@tcp103 [8/256/0/180] [14335.261169] LNet: Added LNI 192.168.203.151@tcp104 [8/256/0/180] [14336.182489] LNet: Added LNI 192.168.203.151@tcp105 [8/256/0/180] [14337.161780] LNet: Added LNI 192.168.203.151@tcp106 [8/256/0/180] [14338.106027] LNet: Added LNI 192.168.203.151@tcp107 [8/256/0/180] [14339.029333] LNet: Added LNI 192.168.203.151@tcp108 [8/256/0/180] [14339.861647] LNet: Added LNI 192.168.203.151@tcp109 [8/256/0/180] [14340.652482] LNet: Added LNI 192.168.203.151@tcp110 [8/256/0/180] [14341.509426] LNet: Added LNI 192.168.203.151@tcp111 [8/256/0/180] [14342.375799] LNet: Added LNI 192.168.203.151@tcp112 [8/256/0/180] [14343.300051] LNet: Added LNI 192.168.203.151@tcp113 [8/256/0/180] [14344.155538] LNet: Added LNI 192.168.203.151@tcp114 [8/256/0/180] [14345.025982] LNet: Added LNI 192.168.203.151@tcp115 [8/256/0/180] [14345.938303] LNet: Added LNI 192.168.203.151@tcp116 [8/256/0/180] [14346.835600] LNet: Added LNI 192.168.203.151@tcp117 [8/256/0/180] [14347.650898] LNet: Added LNI 192.168.203.151@tcp118 [8/256/0/180] [14348.494629] LNet: Added LNI 192.168.203.151@tcp119 [8/256/0/180] [14349.288639] LNet: Added LNI 192.168.203.151@tcp120 [8/256/0/180] [14350.235143] LNet: Added LNI 192.168.203.151@tcp121 [8/256/0/180] [14351.249843] LNet: Added LNI 192.168.203.151@tcp122 [8/256/0/180] [14352.131780] LNet: Added LNI 192.168.203.151@tcp123 [8/256/0/180] [14352.950744] LNet: Added LNI 192.168.203.151@tcp124 [8/256/0/180] [14353.759255] LNet: Added LNI 192.168.203.151@tcp125 [8/256/0/180] [14354.668317] LNet: Added LNI 192.168.203.151@tcp126 [8/256/0/180] [14355.631652] LNet: Added LNI 192.168.203.151@tcp127 [8/256/0/180] [14356.532606] LNet: Added LNI 192.168.203.151@tcp128 [8/256/0/180] [14357.487480] LNet: Added LNI 192.168.203.151@tcp129 [8/256/0/180] [14358.272918] LNet: Added LNI 192.168.203.151@tcp130 [8/256/0/180] [14359.219565] LNet: Added LNI 192.168.203.151@tcp131 [8/256/0/180] [14360.026975] LNet: Added LNI 192.168.203.151@tcp132 [8/256/0/180] [14360.768299] LNet: Added LNI 192.168.203.151@tcp133 [8/256/0/180] [14361.495464] LNet: Added LNI 192.168.203.151@tcp134 [8/256/0/180] [14362.472142] LNet: Added LNI 192.168.203.151@tcp135 [8/256/0/180] [14369.242630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14369.281114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14370.427327] Lustre: 464016:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.151@tcp: can't add NID 192.168.203.151@tcp130: rc = -75 [14370.516968] LNetError: 464043:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14370.526857] LNetError: Protocol error connecting to 192.168.203.151@tcp120 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14370.539343] LNetError: Skipped 6 previous similar messages [14370.656615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14370.696489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14370.720585] LNet: 7 local NIs in recovery (showing 7): 192.168.203.151@tcp124, 192.168.203.151@tcp125, 192.168.203.151@tcp127, 192.168.203.151@tcp126, 192.168.203.151@tcp128, 192.168.203.151@tcp120, 192.168.203.151@tcp129 [14370.834967] Lustre: lustre-MDT0000: new disk, initializing [14370.939471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14370.959713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14371.554951] LNetError: 461475:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14371.570895] LNetError: 461475:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14371.576854] LNetError: Protocol error connecting to 192.168.203.151@tcp100 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14371.591699] LNetError: Skipped 4 previous similar messages [14373.858592] LNetError: 461475:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14373.878906] LNetError: 461475:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14373.902204] LNetError: Protocol error connecting to 192.168.203.151@tcp127 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14373.925108] LNetError: Skipped 9 previous similar messages [14375.271869] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14377.960136] LNetError: 464046:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14377.976048] LNetError: 464046:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14377.997825] LNetError: Protocol error connecting to 192.168.203.151@tcp127 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14386.035035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14386.159866] Lustre: 464034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14386.223623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14386.227628] Lustre: Skipped 1 previous similar message [14386.287930] Lustre: lustre-MDT0001: new disk, initializing [14386.341292] Lustre: 464836:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.151@tcp: can't add NID 192.168.203.151@tcp130: rc = -75 [14386.356201] Lustre: 464836:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14386.539968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14386.595936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14386.610684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14390.900160] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14400.993296] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14401.442923] LNetError: 461477:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14401.471294] LNetError: Protocol error connecting to 192.168.203.151@tcp123 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14406.508233] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14409.647768] LNetError: 461474:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14409.659877] LNetError: 461474:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14409.666942] LNetError: Protocol error connecting to 192.168.203.151@tcp122 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14409.682181] LNetError: Skipped 7 previous similar messages [14413.308760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14413.604736] Lustre: lustre-OST0000: new disk, initializing [14413.610393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14413.618257] Lustre: 466009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14413.714881] Lustre: 466014:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.151@tcp: can't add NID 192.168.203.151@tcp130: rc = -75 [14413.727976] Lustre: 466014:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14413.782729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14420.489314] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14423.575606] Lustre: 466488:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.151@tcp: can't add NID 192.168.203.151@tcp130: rc = -75 [14423.591796] Lustre: 466488:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14423.650741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14423.658538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14423.755023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14426.019093] LNetError: 464037:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14426.032695] LNetError: 464037:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [14426.038530] LNetError: Protocol error connecting to 192.168.203.151@tcp123 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14426.043627] LNetError: Skipped 12 previous similar messages [14429.151254] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14431.201219] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp127 [14439.397469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14439.405158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14439.441180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14442.842936] Lustre: server umount lustre-OST0000 complete [14448.097716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14448.114407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14448.123594] Lustre: Skipped 1 previous similar message [14448.128367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14448.135421] Lustre: Skipped 1 previous similar message [14449.645855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14449.646038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14453.419925] Lustre: server umount lustre-MDT0000 complete [14457.478412] LustreError: 464027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524384 with bad export cookie 9865395555310141668 [14457.488870] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14457.491796] LustreError: 464027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14457.935888] Lustre: server umount lustre-MDT0001 complete [14465.962180] LNetError: 461476:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14465.978654] LNetError: 461476:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [14465.991691] LNetError: Protocol error connecting to 192.168.203.151@tcp123 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14466.000597] LNetError: Skipped 20 previous similar messages [14467.915559] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [14470.514432] Key type lgssc unregistered [14470.866385] LNet: 467919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14470.872444] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14470.883697] LNet: Removed LNI 192.168.203.151@tcp [14471.904781] LNet: Removed LNI 192.168.203.151@tcp100 [14471.910305] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14471.917256] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14472.928195] LNet: Removed LNI 192.168.203.151@tcp101 [14472.931880] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14472.937056] LNet: Removed LNI 192.168.203.151@tcp102 [14473.952610] LNet: Removed LNI 192.168.203.151@tcp103 [14474.976154] LNet: Removed LNI 192.168.203.151@tcp104 [14474.984268] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14474.994796] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14475.007240] LNet: Removed LNI 192.168.203.151@tcp105 [14476.064416] LNet: Removed LNI 192.168.203.151@tcp106 [14477.090636] LNet: Removed LNI 192.168.203.151@tcp107 [14478.112201] LNet: Removed LNI 192.168.203.151@tcp108 [14479.136163] LNet: Removed LNI 192.168.203.151@tcp109 [14479.146265] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14479.156443] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14479.164548] LNet: Removed LNI 192.168.203.151@tcp110 [14480.227403] LNet: Removed LNI 192.168.203.151@tcp111 [14481.248171] LNet: Removed LNI 192.168.203.151@tcp112 [14482.272364] LNet: Removed LNI 192.168.203.151@tcp113 [14483.296766] LNet: Removed LNI 192.168.203.151@tcp114 [14484.322090] LNet: Removed LNI 192.168.203.151@tcp115 [14485.345608] LNet: Removed LNI 192.168.203.151@tcp116 [14486.368754] LNet: Removed LNI 192.168.203.151@tcp117 [14487.392514] LNet: Removed LNI 192.168.203.151@tcp118 [14487.394251] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14487.405971] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14487.419134] LNet: Removed LNI 192.168.203.151@tcp119 [14488.481485] LNet: Removed LNI 192.168.203.151@tcp120 [14489.504137] LNet: Removed LNI 192.168.203.151@tcp121 [14490.528151] LNet: Removed LNI 192.168.203.151@tcp122 [14491.552517] LNet: Removed LNI 192.168.203.151@tcp123 [14492.576198] LNet: Removed LNI 192.168.203.151@tcp124 [14493.600282] LNet: Removed LNI 192.168.203.151@tcp125 [14494.624398] LNet: Removed LNI 192.168.203.151@tcp126 [14494.630560] LNet: Removed LNI 192.168.203.151@tcp127 [14495.649279] LNet: Removed LNI 192.168.203.151@tcp128 [14496.674377] LNet: Removed LNI 192.168.203.151@tcp129 [14496.690323] LNet: Removed LNI 192.168.203.151@tcp130 [14496.703818] LNet: Removed LNI 192.168.203.151@tcp131 [14496.713190] LNet: Removed LNI 192.168.203.151@tcp132 [14496.716292] LNet: Removed LNI 192.168.203.151@tcp133 [14496.719806] LNet: Removed LNI 192.168.203.151@tcp134 [14496.736602] LNet: Removed LNI 192.168.203.151@tcp135 [14497.379606] Key type .llcrypt unregistered [14497.381156] Key type ._llcrypt unregistered [14519.357822] Key type ._llcrypt registered [14519.360231] Key type .llcrypt registered [14519.511990] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [14533.503187] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14535.037355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14535.071707] alg: No test for adler32 (adler32-zlib) [14536.150116] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [14536.357264] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [14538.096173] Key type lgssc registered [14539.208366] Lustre: Echo OBD driver; http://www.lustre.org/ [14575.069479] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:21:39 (1788524499) [14599.381919] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [14602.746539] Key type lgssc unregistered [14603.174283] LNet: 472007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14603.183291] LNetError: 472007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14603.207166] LNet: Removed LNI 192.168.203.151@tcp [14604.271795] Key type .llcrypt unregistered [14604.276357] Key type ._llcrypt unregistered [14619.894379] Key type ._llcrypt registered [14619.895892] Key type .llcrypt registered [14619.987949] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14620.915351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14620.932948] alg: No test for adler32 (adler32-zlib) [14622.013258] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [14622.233578] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [14623.952242] Key type lgssc registered [14625.087902] Lustre: Echo OBD driver; http://www.lustre.org/ [14633.895718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14633.919991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14635.141424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14635.170840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14635.226405] Lustre: lustre-MDT0000: new disk, initializing [14635.279151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14635.291471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14639.605200] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14650.071163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14650.153933] Lustre: 473214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14650.201382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14650.205536] Lustre: Skipped 1 previous similar message [14650.262313] Lustre: lustre-MDT0001: new disk, initializing [14650.341104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14650.377937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14650.392972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14654.375645] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14662.328850] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14667.908141] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14674.353416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14674.700757] Lustre: lustre-OST0000: new disk, initializing [14674.708616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14674.720042] Lustre: 475182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14674.833733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14680.965136] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14683.242674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14683.253211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14683.369563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14690.227302] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14694.652288] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing lnet_if_list [14700.590645] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14700.634852] LNet: Added LNI 192.168.203.151@tcp10 [8/256/0/180] [14706.348363] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14706.371861] LNet: Added LNI 192.168.203.151@tcp11 [8/256/0/180] [14710.367964] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14710.395413] LNet: Added LNI 192.168.203.151@tcp12 [8/256/0/180] [14715.253139] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14715.296720] LNet: Added LNI 192.168.203.151@tcp13 [8/256/0/180] [14720.882928] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14720.935455] LNet: Added LNI 192.168.203.151@tcp14 [8/256/0/180] [14728.953620] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14729.001727] LNet: Added LNI 192.168.203.151@tcp15 [8/256/0/180] [14734.561965] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14734.607250] LNet: Added LNI 192.168.203.151@tcp16 [8/256/0/180] [14740.497890] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14740.532399] LNet: Added LNI 192.168.203.151@tcp17 [8/256/0/180] [14745.573257] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14745.609110] LNet: Added LNI 192.168.203.151@tcp18 [8/256/0/180] [14750.236116] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14750.271633] LNet: Added LNI 192.168.203.151@tcp19 [8/256/0/180] [14754.437828] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14754.469702] LNet: Added LNI 192.168.203.151@tcp20 [8/256/0/180] [14759.269469] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14759.292510] LNet: Added LNI 192.168.203.151@tcp21 [8/256/0/180] [14764.682956] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14764.702866] LNet: Added LNI 192.168.203.151@tcp22 [8/256/0/180] [14770.015722] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14770.041030] LNet: Added LNI 192.168.203.151@tcp23 [8/256/0/180] [14774.940360] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14774.972732] LNet: Added LNI 192.168.203.151@tcp24 [8/256/0/180] [14780.495101] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14780.538843] LNet: Added LNI 192.168.203.151@tcp25 [8/256/0/180] [14785.477247] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14785.504403] LNet: Added LNI 192.168.203.151@tcp26 [8/256/0/180] [14790.461830] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14790.506740] LNet: Added LNI 192.168.203.151@tcp27 [8/256/0/180] [14796.213544] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14796.270416] LNet: Added LNI 192.168.203.151@tcp28 [8/256/0/180] [14802.085604] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14802.119509] LNet: Added LNI 192.168.203.151@tcp29 [8/256/0/180] [14807.947360] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14807.981428] LNet: Added LNI 192.168.203.151@tcp30 [8/256/0/180] [14814.235719] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14814.273365] LNet: Added LNI 192.168.203.151@tcp31 [8/256/0/180] [14820.365528] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14820.389849] LNet: Added LNI 192.168.203.151@tcp32 [8/256/0/180] [14826.333225] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14826.361556] LNet: Added LNI 192.168.203.151@tcp33 [8/256/0/180] [14832.848903] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14832.881486] LNet: Added LNI 192.168.203.151@tcp34 [8/256/0/180] [14839.399698] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14839.424781] LNet: Added LNI 192.168.203.151@tcp35 [8/256/0/180] [14846.221843] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14846.275470] LNet: Added LNI 192.168.203.151@tcp36 [8/256/0/180] [14853.836780] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14853.892355] LNet: Added LNI 192.168.203.151@tcp37 [8/256/0/180] [14860.663600] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14860.698371] LNet: Added LNI 192.168.203.151@tcp38 [8/256/0/180] [14867.192877] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14867.223771] LNet: Added LNI 192.168.203.151@tcp39 [8/256/0/180] [14874.098398] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14874.139931] LNet: Added LNI 192.168.203.151@tcp40 [8/256/0/180] [14880.736298] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14880.764567] LNet: Added LNI 192.168.203.151@tcp41 [8/256/0/180] [14886.423664] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14886.452912] LNet: Added LNI 192.168.203.151@tcp42 [8/256/0/180] [14893.298601] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14893.322884] LNet: Added LNI 192.168.203.151@tcp43 [8/256/0/180] [14900.423779] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14900.477764] LNet: Added LNI 192.168.203.151@tcp44 [8/256/0/180] [14908.601917] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14908.634605] LNet: Added LNI 192.168.203.151@tcp45 [8/256/0/180] [14916.258922] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14916.295398] LNet: Added LNI 192.168.203.151@tcp46 [8/256/0/180] [14924.208966] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14924.267098] LNet: Added LNI 192.168.203.151@tcp47 [8/256/0/180] [14932.256807] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14932.292077] LNet: Added LNI 192.168.203.151@tcp48 [8/256/0/180] [14939.959471] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14939.988527] LNet: Added LNI 192.168.203.151@tcp49 [8/256/0/180] [14948.018658] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14948.058794] LNet: Added LNI 192.168.203.151@tcp50 [8/256/0/180] [14967.777646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14967.786144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14967.805668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14970.349317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14970.349408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14970.372127] Lustre: Skipped 1 previous similar message [14970.391041] Lustre: Skipped 2 previous similar messages [14971.836912] Lustre: server umount lustre-MDT0000 complete [14976.126930] LustreError: 474009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788524902 with bad export cookie 9145760006358565183 [14976.129336] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14976.136668] LustreError: 474009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14976.619378] Lustre: server umount lustre-MDT0001 complete [14996.965447] Lustre: 472488:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788524907/real 1788524907] req@00000000fa2b4616 x1875403916608512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788524923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14997.010269] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14997.980551] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14997.986135] Lustre: 472486:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788524908/real 1788524908] req@000000008d1c1bfa x1875403916608640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788524924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15002.144266] Lustre: 472486:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788524912/real 1788524912] req@000000004f7b73e0 x1875403916608768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788524928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15007.023352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15007.201349] Lustre: 472488:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788524917/real 1788524917] req@000000009477b70a x1875403916609024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788524933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15012.320272] Lustre: 472487:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788524922/real 1788524922] req@00000000a18fdbd7 x1875403916609152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788524938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15020.579253] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7eec4444fbfa0b72 [15020.589389] Lustre: MGC192.168.203.151@tcp: Connection restored to 0@lo (at 0@lo) [15020.902023] LustreError: 475176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15020.922286] LustreError: 475176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15020.987626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15021.995852] LustreError: 475177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15022.029118] LustreError: 475177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15025.560233] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15026.093765] LustreError: 475176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15026.102900] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15026.118566] LustreError: 475176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15031.268756] LustreError: 499419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15031.298979] LustreError: 499419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15033.907072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15034.174524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15038.839642] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15039.479477] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15046.569462] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15051.458251] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15070.179971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15070.183650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15070.202613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15074.955297] Lustre: server umount lustre-OST0000 complete [15080.417802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15080.419405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15080.434152] Lustre: Skipped 1 previous similar message [15080.442557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15080.442565] Lustre: Skipped 1 previous similar message [15084.907273] Lustre: server umount lustre-MDT0000 complete [15085.536986] LustreError: 499418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15085.568864] LustreError: 499418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15089.174201] LustreError: 473205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525015 with bad export cookie 9145760006358567794 [15089.175274] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15089.180127] LustreError: 473205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15089.688424] Lustre: server umount lustre-MDT0001 complete [15101.632612] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [15103.912669] Key type lgssc unregistered [15104.296247] LNet: 502199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15104.310192] LNetError: 502199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15104.315049] LNet: Removed LNI 192.168.203.151@tcp [15104.316753] LNet: Removed LNI 192.168.203.151@tcp10 [15104.320947] LNet: Removed LNI 192.168.203.151@tcp11 [15104.325844] LNet: Removed LNI 192.168.203.151@tcp12 [15104.329733] LNet: Removed LNI 192.168.203.151@tcp13 [15104.338586] LNet: Removed LNI 192.168.203.151@tcp14 [15104.345454] LNet: Removed LNI 192.168.203.151@tcp15 [15104.348641] LNet: Removed LNI 192.168.203.151@tcp16 [15104.350694] LNet: Removed LNI 192.168.203.151@tcp17 [15104.352630] LNet: Removed LNI 192.168.203.151@tcp18 [15104.354432] LNet: Removed LNI 192.168.203.151@tcp19 [15104.362428] LNet: Removed LNI 192.168.203.151@tcp20 [15104.365641] LNet: Removed LNI 192.168.203.151@tcp21 [15104.368819] LNet: Removed LNI 192.168.203.151@tcp22 [15104.371682] LNet: Removed LNI 192.168.203.151@tcp23 [15104.375261] LNet: Removed LNI 192.168.203.151@tcp24 [15104.378462] LNet: Removed LNI 192.168.203.151@tcp25 [15104.381070] LNet: Removed LNI 192.168.203.151@tcp26 [15104.388202] LNet: Removed LNI 192.168.203.151@tcp27 [15104.391574] LNet: Removed LNI 192.168.203.151@tcp28 [15104.396957] LNet: Removed LNI 192.168.203.151@tcp29 [15104.399621] LNet: Removed LNI 192.168.203.151@tcp30 [15104.402241] LNet: Removed LNI 192.168.203.151@tcp31 [15104.404285] LNet: Removed LNI 192.168.203.151@tcp32 [15104.407809] LNet: Removed LNI 192.168.203.151@tcp33 [15104.412108] LNet: Removed LNI 192.168.203.151@tcp34 [15104.416228] LNet: Removed LNI 192.168.203.151@tcp35 [15104.422497] LNet: Removed LNI 192.168.203.151@tcp36 [15104.426412] LNet: Removed LNI 192.168.203.151@tcp37 [15104.429821] LNet: Removed LNI 192.168.203.151@tcp38 [15104.432663] LNet: Removed LNI 192.168.203.151@tcp39 [15104.436184] LNet: Removed LNI 192.168.203.151@tcp40 [15104.439239] LNet: Removed LNI 192.168.203.151@tcp41 [15104.442725] LNet: Removed LNI 192.168.203.151@tcp42 [15104.447571] LNet: Removed LNI 192.168.203.151@tcp43 [15104.451106] LNet: Removed LNI 192.168.203.151@tcp44 [15104.455447] LNet: Removed LNI 192.168.203.151@tcp45 [15104.459303] LNet: Removed LNI 192.168.203.151@tcp46 [15104.463226] LNet: Removed LNI 192.168.203.151@tcp47 [15104.465207] LNet: Removed LNI 192.168.203.151@tcp48 [15104.469263] LNet: Removed LNI 192.168.203.151@tcp49 [15104.478610] LNet: Removed LNI 192.168.203.151@tcp50 [15105.117405] Key type .llcrypt unregistered [15105.121259] Key type ._llcrypt unregistered [15124.850681] Key type ._llcrypt registered [15124.852810] Key type .llcrypt registered [15124.958736] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [15139.920269] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15141.049091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15141.091777] alg: No test for adler32 (adler32-zlib) [15142.220229] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [15142.487406] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [15144.184265] Key type lgssc registered [15145.203628] Lustre: Echo OBD driver; http://www.lustre.org/ [15179.322438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15179.361211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15180.608351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15180.645202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15180.707292] Lustre: lustre-MDT0000: new disk, initializing [15180.796632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15180.814392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15184.756718] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15194.990275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15195.128242] Lustre: 505393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15195.190280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15195.194270] Lustre: Skipped 1 previous similar message [15195.253082] Lustre: lustre-MDT0001: new disk, initializing [15195.351614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15195.395226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15195.417296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15200.536714] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15210.529423] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15216.140286] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15224.341490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15224.613908] Lustre: *** cfs_fail_loc=90f, val=0*** [15224.734119] Lustre: lustre-OST0000: new disk, initializing [15224.738226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15224.747394] Lustre: 507411:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15224.848185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15230.705762] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15232.575130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15232.586757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15232.699208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15240.245469] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15248.371722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15248.373965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15248.374742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15248.394945] Lustre: Skipped 1 previous similar message [15248.426280] LustreError: Skipped 1 previous similar message [15253.643942] Lustre: server umount lustre-OST0000 complete [15258.593811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15258.607179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15258.614320] Lustre: Skipped 2 previous similar messages [15263.719162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15268.836036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15268.841085] Lustre: Skipped 2 previous similar messages [15271.394630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15271.677477] Lustre: server umount lustre-MDT0000 complete [15273.958368] LustreError: 505404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15273.985493] LustreError: 505404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15275.188250] LustreError: 507410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525201 with bad export cookie 15337762283281388430 [15275.194567] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15275.199607] LustreError: 507410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15275.662342] Lustre: server umount lustre-MDT0001 complete [15287.386621] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [15290.360918] Key type lgssc unregistered [15290.729910] LNet: 509316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15290.748082] LNetError: 509316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15290.758849] LNet: Removed LNI 192.168.203.151@tcp [15291.639577] Key type .llcrypt unregistered [15291.644420] Key type ._llcrypt unregistered [15311.128529] Key type ._llcrypt registered [15311.131179] Key type .llcrypt registered [15311.232904] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15312.646851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15312.697906] alg: No test for adler32 (adler32-zlib) [15313.927225] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [15314.272159] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [15315.984365] Key type lgssc registered [15317.352978] Lustre: Echo OBD driver; http://www.lustre.org/ [15324.765251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15324.784973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15326.278961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15330.793194] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15336.419753] LustreError: 510585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15336.446948] LustreError: 510585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15339.530770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15339.920609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15344.930286] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15352.465916] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15357.486732] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15364.402762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15364.532889] LustreError: 512406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15364.766293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15370.787611] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15379.015528] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15392.935155] loop0: detected capacity change from 0 to 204800000 [15393.245162] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15393.257339] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15393.279170] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15400.939417] loop0: detected capacity change from 0 to 204800000 [15401.273754] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15401.288438] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15401.298776] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15408.293826] loop0: detected capacity change from 0 to 204800000 [15415.619477] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15415.765550] Lustre: fs15246-OST0000: new disk, initializing [15415.768744] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15415.774631] Lustre: 514180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15415.838880] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15422.898849] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15433.128412] loop1: detected capacity change from 0 to 204800000 [15439.482838] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15439.691800] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15439.760233] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15439.836078] Lustre: fs15246-MDT0000: new disk, initializing [15439.988308] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15440.033514] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15440.184618] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15441.253848] Lustre: 515302:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000546016c3 x1875404642107136/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788525383 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15444.506351] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15450.346323] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15450.359316] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15450.382843] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15468.024043] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15468.055834] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15470.402464] Lustre: server umount fs15246-MDT0000 complete [15473.134934] LustreError: 512019:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15478.250222] LustreError: 512019:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15480.508137] Lustre: server umount fs15246-OST0000 complete [15493.089666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15493.098520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15493.119381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15498.725467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15498.735415] Lustre: Skipped 2 previous similar messages [15503.854808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15503.864823] Lustre: Skipped 1 previous similar message [15506.400301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15506.626347] Lustre: server umount lustre-OST0000 complete [15514.082459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15514.085250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15514.097660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15514.104842] Lustre: Skipped 2 previous similar messages [15514.123324] Lustre: Skipped 1 previous similar message [15516.646888] Lustre: server umount lustre-MDT0000 complete [15519.209916] LustreError: 511313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15519.247208] LustreError: 511313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15520.402114] LustreError: 510565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525446 with bad export cookie 8698914603317455977 [15520.403972] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15520.422152] LustreError: 510565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15520.727083] Lustre: server umount lustre-MDT0001 complete [15533.187972] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [15536.023907] Key type lgssc unregistered [15536.369600] LNet: 518399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15536.384899] LNetError: 518399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15536.405398] LNet: Removed LNI 192.168.203.151@tcp [15537.481701] Key type .llcrypt unregistered [15537.484638] Key type ._llcrypt unregistered [15555.406563] Key type ._llcrypt registered [15555.408763] Key type .llcrypt registered [15555.513180] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15556.568623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15556.703965] alg: No test for adler32 (adler32-zlib) [15557.766889] Lustre: Lustre: Build Version: 2.17.58_4_g439d16b [15558.024333] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [15559.696553] Key type lgssc registered [15560.795889] Lustre: Echo OBD driver; http://www.lustre.org/ [15597.378423] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [15608.550275] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15646.053972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15646.081199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15647.422518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15647.464870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15647.549361] Lustre: lustre-MDT0000: new disk, initializing [15647.679539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15647.718179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15651.816813] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15661.804706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15662.021736] Lustre: 522863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15662.106249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15662.111289] Lustre: Skipped 1 previous similar message [15662.196714] Lustre: lustre-MDT0001: new disk, initializing [15662.316495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15662.375803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15662.393732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15667.031171] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15676.468229] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15681.759572] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15688.636586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15688.925268] Lustre: lustre-OST0000: new disk, initializing [15688.938535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15688.949698] Lustre: 524830:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15689.068649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15695.425851] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15695.433043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15695.456501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15695.547485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15706.347937] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15711.633596] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15711.868157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15716.721813] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15716.956117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15721.455451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15721.458326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15721.463891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15721.499210] Lustre: Skipped 1 previous similar message [15726.564580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15726.574666] Lustre: Skipped 2 previous similar messages [15731.695401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15731.704918] Lustre: Skipped 1 previous similar message [15733.216178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15733.433134] Lustre: server umount lustre-OST0000 complete [15739.363912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15739.377084] LustreError: Skipped 1 previous similar message [15739.381226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15739.392517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15741.923497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15743.270596] Lustre: server umount lustre-MDT0000 complete [15746.852777] LustreError: 522854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788525673 with bad export cookie 17650924883651593835 [15746.854227] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15746.861859] LustreError: 522854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15747.188964] Lustre: server umount lustre-MDT0001 complete [15770.604927] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [15780.882746] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15812.774544] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15511 sec ========= 08:42:17 (1788525737) [15814.299243] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:42:18 (1788525738) === [15817.874663] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:42:22 (1788525742) === [15857.665799] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [15860.293419] Key type lgssc unregistered [15860.618298] LNet: 531472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15860.637982] LNetError: 531472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15860.654409] LNet: Removed LNI 192.168.203.151@tcp [15861.526951] Key type .llcrypt unregistered [15861.529043] Key type ._llcrypt unregistered