[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491873131 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003216] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007792] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011152] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022645] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027366] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028181] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030136] *** VALIDATE tmpfs *** [ 0.031495] *** VALIDATE proc *** [ 0.033265] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.036248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038072] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.043909] debug: unmapping init [mem 0xffffffff93259000-0xffffffff93260fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046774] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051021] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054319] rcu: Hierarchical SRCU implementation. [ 0.056597] smp: Bringing up secondary CPUs ... [ 0.057627] x86: Booting SMP configuration: [ 0.058031] .... node #0, CPUs: #1 #2 #3 [ 0.062111] smp: Brought up 1 node, 4 CPUs [ 0.064021] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.206781] node 0 deferred pages initialised in 140ms [ 0.210016] devtmpfs: initialized [ 0.211343] x86/mm: Memory block size: 128MB [ 0.213996] gcov: version magic: 0x41383552 [ 0.216190] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.217093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.218264] pinctrl core: initialized pinctrl subsystem [ 0.219206] [ 0.219850] ************************************************************* [ 0.220024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221035] ** ** [ 0.222019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.223016] ** ** [ 0.224018] ** This means that this kernel is built to expose internal ** [ 0.225018] ** IOMMU data structures, which may compromise security on ** [ 0.226018] ** your system. ** [ 0.227017] ** ** [ 0.228018] ** If you see this message and you are not debugging the ** [ 0.229019] ** kernel, report this immediately to your vendor! ** [ 0.230022] ** ** [ 0.231016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.232031] ************************************************************* [ 0.233848] NET: Registered protocol family 16 [ 0.234495] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.235072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.236083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.237451] cpuidle: using governor menu [ 0.239370] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.241808] PCI: Using configuration type 1 for base access [ 0.243154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.254120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.256024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.260113] cryptd: max_cpu_qlen set to 1000 [ 0.262237] ACPI: Added _OSI(Module Device) [ 0.263024] ACPI: Added _OSI(Processor Device) [ 0.264019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.265018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.269040] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.272531] ACPI: Interpreter enabled [ 0.273075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.274016] ACPI: Using IOAPIC for interrupt routing [ 0.275109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.276647] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.285555] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.286053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.287025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290451] acpiphp: Slot [2] registered [ 0.291000] acpiphp: Slot [5] registered [ 0.293192] acpiphp: Slot [6] registered [ 0.295257] acpiphp: Slot [7] registered [ 0.297174] acpiphp: Slot [8] registered [ 0.298136] acpiphp: Slot [9] registered [ 0.300110] acpiphp: Slot [10] registered [ 0.301217] acpiphp: Slot [3] registered [ 0.303118] acpiphp: Slot [4] registered [ 0.304232] acpiphp: Slot [11] registered [ 0.306178] acpiphp: Slot [12] registered [ 0.307133] acpiphp: Slot [13] registered [ 0.309210] acpiphp: Slot [14] registered [ 0.310113] acpiphp: Slot [15] registered [ 0.312094] acpiphp: Slot [16] registered [ 0.313085] acpiphp: Slot [17] registered [ 0.314177] acpiphp: Slot [18] registered [ 0.315196] acpiphp: Slot [19] registered [ 0.316130] acpiphp: Slot [20] registered [ 0.318209] acpiphp: Slot [21] registered [ 0.319127] acpiphp: Slot [22] registered [ 0.321136] acpiphp: Slot [23] registered [ 0.323118] acpiphp: Slot [24] registered [ 0.324153] acpiphp: Slot [25] registered [ 0.325121] acpiphp: Slot [26] registered [ 0.327122] acpiphp: Slot [27] registered [ 0.328114] acpiphp: Slot [28] registered [ 0.330118] acpiphp: Slot [29] registered [ 0.331104] acpiphp: Slot [30] registered [ 0.332101] acpiphp: Slot [31] registered [ 0.334091] PCI host bridge to bus 0000:00 [ 0.335016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.339025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351161] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354245] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362750] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367956] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377043] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379515] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382577] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386047] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398991] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.406233] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.423023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.445125] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.455025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.475020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.484235] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.496029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.509024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.518465] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.524019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.529022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.542021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.553060] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.560015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.564016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.580021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.592868] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.601029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.609023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.630028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.642932] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.645516] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.648477] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.650377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.652259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.657051] iommu: Default domain type: Passthrough [ 0.660425] SCSI subsystem initialized [ 0.661173] ACPI: bus type USB registered [ 0.663120] usbcore: registered new interface driver usbfs [ 0.664088] usbcore: registered new interface driver hub [ 0.665116] usbcore: registered new device driver usb [ 0.667181] pps_core: LinuxPPS API ver. 1 registered [ 0.668010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.670079] PTP clock support registered [ 0.672219] EDAC MC: Ver: 3.0.0 [ 0.674144] PCI: Using ACPI for IRQ routing [ 0.677121] NetLabel: Initializing [ 0.679057] NetLabel: domain hash size = 128 [ 0.680011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.682113] NetLabel: unlabeled traffic allowed by default [ 0.683163] vgaarb: loaded [ 0.684303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.686011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.696082] clocksource: Switched to clocksource kvm-clock [ 0.806215] VFS: Disk quotas dquot_6.6.0 [ 0.807598] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.810229] *** VALIDATE ramfs *** [ 0.811468] *** VALIDATE hugetlbfs *** [ 0.813688] pnp: PnP ACPI init [ 0.816127] pnp: PnP ACPI: found 6 devices [ 0.832071] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.835387] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.837689] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.839845] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.842215] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.844564] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.847355] NET: Registered protocol family 2 [ 0.849732] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.854615] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.858134] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.863309] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.867808] TCP: Hash tables configured (established 65536 bind 65536) [ 0.870268] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.872754] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.875712] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.879181] NET: Registered protocol family 1 [ 0.882372] RPC: Registered named UNIX socket transport module. [ 0.883637] RPC: Registered udp transport module. [ 0.884805] RPC: Registered tcp transport module. [ 0.885709] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.887081] NET: Registered protocol family 44 [ 0.888779] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.890800] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.893147] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.895352] PCI: CLS 0 bytes, default 64 [ 0.897368] Unpacking initramfs... [ 2.289944] debug: unmapping init [mem 0xffff9755fcc54000-0xffff9755fffbffff] [ 2.294031] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.296360] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.298954] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.796153] Initialise system trusted keyrings [ 2.797457] Key type blacklist registered [ 2.799072] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.807406] zbud: loaded [ 2.810197] *** VALIDATE nfs *** [ 2.811182] *** VALIDATE nfs4 *** [ 2.812420] pstore: using deflate compression [ 2.815349] Platform Keyring initialized [ 2.930593] NET: Registered protocol family 38 [ 2.932528] Key type asymmetric registered [ 2.933665] Asymmetric key parser 'x509' registered [ 2.934979] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.938218] io scheduler mq-deadline registered [ 2.939486] io scheduler kyber registered [ 2.941082] io scheduler bfq registered [ 2.942541] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.944958] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.948106] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.951620] ACPI: Power Button [PWRF] [ 2.957429] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.966038] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.985835] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.995577] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.024174] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.055449] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.086716] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.090609] Non-volatile memory driver v1.3 [ 3.091919] Linux agpgart interface v0.103 [ 3.119971] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.122066] vda: detected capacity change from 0 to 76775424 [ 3.136380] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.138562] vdb: detected capacity change from 0 to 1073741824 [ 3.150972] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.153098] vdc: detected capacity change from 0 to 2621440000 [ 3.165356] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.167545] vdd: detected capacity change from 0 to 2621440000 [ 3.181020] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.183206] vde: detected capacity change from 0 to 4294967296 [ 3.195227] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.198044] vdf: detected capacity change from 0 to 4294967296 [ 3.204398] libphy: Fixed MDIO Bus: probed [ 3.211374] usbcore: registered new interface driver usbserial_generic [ 3.213213] usbserial: USB Serial support registered for generic [ 3.214917] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.218470] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.219838] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.221805] mousedev: PS/2 mouse device common for all mice [ 3.224712] rtc_cmos 00:05: RTC can wake from S4 [ 3.227437] rtc_cmos 00:05: registered as rtc0 [ 3.228781] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.231033] intel_pstate: CPU model not supported [ 3.233286] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.236919] hid: raw HID events driver (C) Jiri Kosina [ 3.238810] usbcore: registered new interface driver usbhid [ 3.240225] usbhid: USB HID core driver [ 3.241650] drop_monitor: Initializing network drop monitor service [ 3.243581] Initializing XFRM netlink socket [ 3.245820] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.245861] NET: Registered protocol family 10 [ 3.251162] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.252044] Segment Routing with IPv6 [ 3.254346] NET: Registered protocol family 17 [ 3.256173] mpls_gso: MPLS GSO support [ 3.261360] RAS: Correctable Errors collector initialized. [ 3.263097] AVX version of gcm_enc/dec engaged. [ 3.264254] AES CTR mode by8 optimization enabled [ 3.355941] sched_clock: Marking stable (3355909277, 0)->(4334781513, -978872236) [ 3.359490] registered taskstats version 1 [ 3.362139] Loading compiled-in X.509 certificates [ 3.363866] zswap: loaded using pool lzo/zbud [ 3.390860] Key type big_key registered [ 3.405554] Key type encrypted registered [ 3.406949] ima: No TPM chip found, activating TPM-bypass! [ 3.408884] ima: Allocated hash algorithm: sha1 [ 3.410848] ima: No architecture policies found [ 3.412140] evm: Initialising EVM extended attributes: [ 3.413488] evm: security.selinux [ 3.414546] evm: security.ima [ 3.415491] evm: security.capability [ 3.416540] evm: HMAC attrs: 0x1 [ 3.419259] rtc_cmos 00:05: setting system clock to 2026-09-17 19:30:18 UTC (1789673418) [ 3.425751] debug: unmapping init [mem 0xffffffff94203000-0xffffffff943fffff] [ 3.428455] debug: unmapping init [mem 0xffffffff92f82000-0xffffffff93258fff] [ 3.436172] Write protecting the kernel read-only data: 28672k [ 3.439415] debug: unmapping init [mem 0xffffffff91603000-0xffffffff917fffff] [ 3.441648] debug: unmapping init [mem 0xffffffff91f14000-0xffffffff91ffffff] [ 3.475323] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.481691] systemd[1]: Detected virtualization kvm. [ 3.483144] systemd[1]: Detected architecture x86-64. [ 3.484819] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.511461] systemd[1]: No hostname configured. [ 3.512757] systemd[1]: Set hostname to . [ 3.514451] random: systemd: uninitialized urandom read (16 bytes read) [ 3.516220] systemd[1]: Initializing machine ID from random generator. [ 3.644311] random: systemd: uninitialized urandom read (16 bytes read) [ 3.646566] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.652270] random: systemd: uninitialized urandom read (16 bytes read) [ 3.654786] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.671235] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.321668] device-mapper: uevent: version 1.0.3 [ 4.323575] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.059674] random: fast init done [ 5.060620] virtio_net virtio0 ens2: renamed from eth0 [ 5.176984] scsi host0: ata_piix [ 5.195900] scsi host1: ata_piix [ 5.197569] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.200340] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.858114] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.921519] random: crng init done [ 9.923107] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.334614] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.430353] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.686719] SELinux: Disabled at runtime. [ 11.753328] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.761811] systemd[1]: Detected virtualization kvm. [ 11.763853] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.228692] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.231657] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.236126] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.240223] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.244227] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.251207] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.259341] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.280564] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.660832] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.881902] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.882714] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.028346] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.042935] EDAC sbridge: Ver: 1.1.2 [ 14.890426] Key type dns_resolver registered [ 15.182532] NFS: Registering the id_resolver key type [ 15.184558] Key type id_resolver registered [ 15.186167] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg149-server login: [ 41.475429] libcfs: loading out-of-tree module taints kernel. [ 41.495892] Key type ._llcrypt registered [ 41.497300] Key type .llcrypt registered [ 41.553746] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 59.121920] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 61.062665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.086759] alg: No test for adler32 (adler32-zlib) [ 62.763877] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 63.857767] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 65.776698] Key type lgssc registered [ 68.114328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 82.436137] hrtimer: interrupt took 7452108 ns [ 90.704198] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 137.137265] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 153.813237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 153.871298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.453624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.508888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.654418] Lustre: lustre-MDT0000: new disk, initializing [ 155.801847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.835861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.461788] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.005875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.117404] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.158833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.163628] Lustre: Skipped 1 previous similar message [ 177.242232] Lustre: lustre-MDT0001: new disk, initializing [ 177.321403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.344580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.352367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.229467] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.678731] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.066049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.365893] Lustre: lustre-OST0000: new disk, initializing [ 198.368441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.392493] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.465605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.927600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.942189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.050673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.921780] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.122399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.253896] Lustre: lustre-OST0001: new disk, initializing [ 219.267293] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.277623] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.399841] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.045622] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.869110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 225.886087] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 225.986924] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 238.610282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.012226] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 256.500383] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing check_logdir /tmp/testlogs/ [ 263.870273] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing yml_node [ 269.182790] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 272.214700] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 274.963844] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 277.209455] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 17 15:34:51 EDT 2026 [ 297.421886] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 299.582980] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 318.435129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.438039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.470908] Lustre: Skipped 1 previous similar message [ 318.492799] Lustre: Skipped 3 previous similar messages [ 322.813848] Lustre: server umount lustre-MDT0000 complete [ 326.736085] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789673741 with bad export cookie 16771702055577117610 [ 326.742670] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 326.752395] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 328.672371] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.673154] LustreError: 6519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.673533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.691195] Lustre: Skipped 3 previous similar messages [ 328.752582] LustreError: 6519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 331.680129] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.683077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.708516] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 331.732849] Lustre: Skipped 2 previous similar messages [ 333.290824] Lustre: server umount lustre-MDT0001 complete [ 337.052274] Lustre: server umount lustre-OST0000 complete [ 340.814079] Lustre: server umount lustre-OST0001 complete [ 348.239929] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 358.542233] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 399.210432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.527237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 399.559328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 399.645249] Lustre: lustre-MDT0000: new disk, initializing [ 399.817175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.855653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 404.630406] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 417.099824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.208682] Lustre: 15095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.299844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.306897] Lustre: Skipped 1 previous similar message [ 417.413398] Lustre: lustre-MDT0001: new disk, initializing [ 417.594355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.676847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 417.712822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 422.866678] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 433.536236] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 439.598269] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.339189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.678357] Lustre: lustre-OST0000: new disk, initializing [ 447.682256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.687558] Lustre: 17064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 447.826647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.646213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.657668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.765123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 454.339404] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 465.189138] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 471.079122] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 471.328420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 476.670623] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 476.871108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.748099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.754237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.768171] LustreError: Skipped 1 previous similar message [ 480.769704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.792288] Lustre: Skipped 1 previous similar message [ 485.264557] Lustre: server umount lustre-OST0000 complete [ 489.441097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 489.452986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.488616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.497729] Lustre: Skipped 1 previous similar message [ 495.828626] Lustre: server umount lustre-MDT0000 complete [ 496.097498] LustreError: 15107:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.126345] LustreError: 15107:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 499.784852] LustreError: 15891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789673914 with bad export cookie 16771702055577119136 [ 499.793328] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.799753] LustreError: 15891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.313894] Lustre: server umount lustre-MDT0001 complete [ 505.124603] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:38:39 (1789673919) [ 507.942232] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 509.779958] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:38:44 (1789673924) [ 516.974045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.504043] LustreError: 19475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.581136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.599241] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 522.720950] LustreError: 19476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.697555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.849471] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 541.360654] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 547.172273] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 553.792027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 561.757655] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 570.600834] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 578.024218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.027092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 578.039457] Lustre: Skipped 1 previous similar message [ 578.041722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 578.059828] Lustre: Skipped 3 previous similar messages [ 583.942416] Lustre: server umount lustre-OST0000 complete [ 588.256885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.286367] Lustre: Skipped 1 previous similar message [ 593.833353] Lustre: server umount lustre-MDT0000 complete [ 597.739173] LustreError: 21968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674012 with bad export cookie 16771702055577120025 [ 597.743752] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.749696] LustreError: 21968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 598.028086] Lustre: server umount lustre-MDT0001 complete [ 609.314908] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 611.923419] Key type lgssc unregistered [ 612.325430] LNet: 23005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.339465] LNetError: 23005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.355227] LNet: Removed LNI 192.168.201.149@tcp [ 613.440150] Key type .llcrypt unregistered [ 613.442734] Key type ._llcrypt unregistered [ 635.923158] Key type ._llcrypt registered [ 635.927409] Key type .llcrypt registered [ 636.116746] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 637.910323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 637.986965] alg: No test for adler32 (adler32-zlib) [ 639.064906] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 639.317966] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 641.017687] Key type lgssc registered [ 642.652774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.436657] loop: module loaded [ 652.512563] loop0: detected capacity change from 0 to 204800000 [ 652.819649] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 652.828498] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 653.251504] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 659.966704] loop0: detected capacity change from 0 to 204800000 [ 660.253256] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.265361] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.274403] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.806310] loop0: detected capacity change from 0 to 204800000 [ 671.061949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 671.106809] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.290322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 672.365359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 672.468770] Lustre: lustre-MDT0000: new disk, initializing [ 672.531525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.546922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 676.667786] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 684.843122] loop1: detected capacity change from 0 to 204800000 [ 690.550847] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 691.154022] Lustre: lustre-OST07c6: new disk, initializing [ 691.163344] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 691.175228] Lustre: Skipped 1 previous similar message [ 691.182987] Lustre: 26192:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 691.287573] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 698.663787] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 699.566242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 699.572277] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 699.612715] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 704.988623] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 704.995799] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 705.001215] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 705.006539] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 705.021884] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 705.036217] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 720.877397] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 720.891600] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 720.898961] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.907416] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 720.917164] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 720.924848] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.967351] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 725.987338] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.996900] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 726.007755] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.017684] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 726.026264] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.036554] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 726.049352] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.059232] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 726.069744] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.080581] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 726.086234] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.183069] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 731.196099] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 731.201985] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 731.208143] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.214643] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 731.225559] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.238935] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 731.242617] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.245693] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 731.248327] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.251814] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 731.257614] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.902421] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 735.908106] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 735.913423] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 735.918917] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.923728] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 735.931791] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.939532] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 735.943856] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.950114] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 735.956304] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.963337] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 735.968819] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.025803] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 744.032356] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 744.035784] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 744.039873] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 744.043845] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 744.047157] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 744.050817] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 744.054315] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 744.057911] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 744.062054] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 744.064548] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.067559] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 796.645093] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.662085] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 801.004203] Lustre: server umount lustre-OST07c6 complete [ 810.291571] Lustre: server umount lustre-MDT0000 complete [ 826.573263] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 829.670709] Key type lgssc unregistered [ 829.925811] LNet: 28746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.939854] LNetError: 28746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.953388] LNet: Removed LNI 192.168.201.149@tcp [ 830.742157] Key type .llcrypt unregistered [ 830.750637] Key type ._llcrypt unregistered [ 851.275132] Key type ._llcrypt registered [ 851.278766] Key type .llcrypt registered [ 851.385623] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 852.700137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 852.783981] alg: No test for adler32 (adler32-zlib) [ 854.263125] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 854.650088] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 856.368189] Key type lgssc registered [ 857.619367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 865.773674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 865.792658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.428348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.694756] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 877.026942] LustreError: 30014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 877.050773] LustreError: 30014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 882.158655] LustreError: 30013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 882.185566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 882.418362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 888.003251] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 896.043736] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 903.592194] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 911.400410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 911.859804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.891326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 919.436365] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 929.276615] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 947.686675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.689934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.703477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.712440] Lustre: Skipped 1 previous similar message [ 952.803230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 952.815922] Lustre: Skipped 1 previous similar message [ 953.531443] Lustre: server umount lustre-OST0000 complete [ 957.921897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.927504] Lustre: Skipped 2 previous similar messages [ 957.929443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 957.933398] Lustre: Skipped 1 previous similar message [ 963.041692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 963.049545] Lustre: Skipped 1 previous similar message [ 964.004433] Lustre: server umount lustre-MDT0000 complete [ 968.164060] LustreError: 30014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.068118] LustreError: 29995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674384 with bad export cookie 7290943347299777002 [ 969.071958] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.097947] LustreError: 29995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 969.501517] Lustre: server umount lustre-MDT0001 complete [ 982.386442] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 985.347897] Key type lgssc unregistered [ 985.758708] LNet: 33753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.768479] LNetError: 33753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.800594] LNet: Removed LNI 192.168.201.149@tcp [ 986.712443] Key type .llcrypt unregistered [ 986.714361] Key type ._llcrypt unregistered [ 1006.256235] Key type ._llcrypt registered [ 1006.258578] Key type .llcrypt registered [ 1006.352378] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1007.605903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1007.642741] alg: No test for adler32 (adler32-zlib) [ 1008.679593] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1008.906081] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1010.567480] Key type lgssc registered [ 1011.654239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1020.413193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1020.429557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.005883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.384570] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1032.162305] LustreError: 35022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.174390] LustreError: 35022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1036.544863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.829102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1040.976562] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1049.424734] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1056.695609] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1063.774594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1064.129554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.192543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1070.468499] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1078.754389] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1092.069297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.083192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1092.098085] Lustre: Skipped 1 previous similar message [ 1092.110428] Lustre: Skipped 1 previous similar message [ 1096.247715] Lustre: server umount lustre-OST0000 complete [ 1102.317164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.334063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1102.347336] Lustre: Skipped 1 previous similar message [ 1102.373221] Lustre: Skipped 1 previous similar message [ 1107.317318] Lustre: server umount lustre-MDT0000 complete [ 1107.433225] LustreError: 35751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1112.545802] LustreError: 35022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1112.575623] LustreError: 35022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1112.745498] LustreError: 37517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674527 with bad export cookie 10429687788504524358 [ 1112.747649] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1112.760779] LustreError: 37517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1113.244355] Lustre: server umount lustre-MDT0001 complete [ 1125.511530] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1128.688039] Key type lgssc unregistered [ 1129.002525] LNet: 38557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.007241] LNetError: 38557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.028790] LNet: Removed LNI 192.168.201.149@tcp [ 1129.718183] Key type .llcrypt unregistered [ 1129.720638] Key type ._llcrypt unregistered [ 1154.384195] Key type ._llcrypt registered [ 1154.385937] Key type .llcrypt registered [ 1154.522486] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1155.435865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1155.516256] alg: No test for adler32 (adler32-zlib) [ 1156.672244] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1156.953522] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1158.615298] Key type lgssc registered [ 1159.882856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1166.556086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1166.573587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.089275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.217967] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1178.592552] LustreError: 39987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.611061] LustreError: 39987:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1180.793547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.105502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1185.121746] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1192.160768] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1197.669417] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1204.358126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.814016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.845751] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1213.040394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1218.159681] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1226.144152] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 1226.889101] Lustre: server umount lustre-MDT0000 complete [ 1227.232936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1227.236938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.249358] LustreError: 39987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.769501] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.772068] LustreError: 39982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.787821] Lustre: Skipped 1 previous similar message [ 1228.806191] LustreError: 39982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1230.040686] LustreError: 39966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674645 with bad export cookie 4130848308526019882 [ 1230.045376] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.058906] LustreError: 39966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1231.240553] LustreError: 39981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.270643] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 1231.279239] LustreError: 39981:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1233.898705] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.914939] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1235.938223] LustreError: 42282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.939308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1235.946445] LustreError: 42282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1236.487942] Lustre: server umount lustre-MDT0001 complete [ 1257.506246] Lustre: server umount lustre-OST0000 complete [ 1274.251255] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1277.056636] Key type lgssc unregistered [ 1277.333044] LNet: 43866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.340744] LNetError: 43866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.364733] LNet: Removed LNI 192.168.201.149@tcp [ 1278.088758] Key type .llcrypt unregistered [ 1278.091516] Key type ._llcrypt unregistered [ 1303.707595] Key type ._llcrypt registered [ 1303.710553] Key type .llcrypt registered [ 1303.805803] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1304.939494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.012411] alg: No test for adler32 (adler32-zlib) [ 1306.200488] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1306.407706] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1308.080107] Key type lgssc registered [ 1309.241350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1316.744830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1316.773637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.198465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.328581] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1328.609562] LustreError: 45280:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.622286] LustreError: 45280:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1331.442356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.683771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.908105] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1342.701162] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1348.238292] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1356.349652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1356.736201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.220195] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1363.884278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1370.922736] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1377.700785] Lustre: lustre-OST0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 1378.273258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1378.280571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.297905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1379.303073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1379.304864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1379.324430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.395471] Lustre: server umount lustre-OST0000 complete [ 1403.364991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1403.373488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.383235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1403.391089] Lustre: Skipped 1 previous similar message [ 1407.591769] Lustre: server umount lustre-MDT0000 complete [ 1409.511692] LustreError: 45281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.532797] LustreError: 45281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1411.847609] LustreError: 45261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789674826 with bad export cookie 6553183081918559401 [ 1411.863630] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1411.870043] LustreError: 45261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1426.400467] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1426.768050] Lustre: server umount lustre-MDT0001 complete [ 1440.123123] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1443.825927] Key type lgssc unregistered [ 1444.168907] LNet: 48993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.176833] LNetError: 48993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.189432] LNet: Removed LNI 192.168.201.149@tcp [ 1445.142792] Key type .llcrypt unregistered [ 1445.144925] Key type ._llcrypt unregistered [ 1471.683685] Key type ._llcrypt registered [ 1471.686298] Key type .llcrypt registered [ 1471.868868] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1473.464179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1473.524747] alg: No test for adler32 (adler32-zlib) [ 1474.596696] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1474.955931] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1476.696063] Key type lgssc registered [ 1478.086708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1487.738805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1487.776880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.198577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.283414] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1499.622517] LustreError: 50408:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.635155] LustreError: 50408:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1504.027986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.388288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1509.337569] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1516.973631] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1523.700272] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1533.039391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1533.704842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.738990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1543.116896] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1552.299170] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1559.390354] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1563.946352] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1576.377092] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1715h56m29s [ 1577.957249] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1580.431896] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 1581.030886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1581.038874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.090719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.178401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.194321] Lustre: Skipped 5 previous similar messages [ 1586.569723] LustreError: 50404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1586.667839] Lustre: server umount lustre-MDT0000 complete [ 1588.721631] LustreError: 51136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1588.741938] LustreError: 51136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1590.717978] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1592.709328] LustreError: 50403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.325190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.580301] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1598.891434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.007749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1599.915775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1603.914216] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1607.979991] Lustre: DEBUG MARKER: Wait for df (36735) ... [ 1609.730557] Lustre: DEBUG MARKER: done [ 1615.331663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.343881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1615.355346] Lustre: Skipped 3 previous similar messages [ 1615.366098] Lustre: Skipped 1 previous similar message [ 1620.449038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1620.467629] Lustre: Skipped 1 previous similar message [ 1620.888614] Lustre: server umount lustre-OST0000 complete [ 1630.689468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.703850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.559782] Lustre: server umount lustre-MDT0000 complete [ 1637.512171] LustreError: 50387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675052 with bad export cookie 9221270051966703976 [ 1637.535761] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1638.006265] Lustre: server umount lustre-MDT0001 complete [ 1653.621341] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1657.928233] Key type lgssc unregistered [ 1658.485084] LNet: 55333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.497581] LNetError: 55333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.528730] LNet: Removed LNI 192.168.201.149@tcp [ 1659.958167] Key type .llcrypt unregistered [ 1659.967468] Key type ._llcrypt unregistered [ 1703.839270] Key type ._llcrypt registered [ 1703.840719] Key type .llcrypt registered [ 1704.024660] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1706.034840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1706.087985] alg: No test for adler32 (adler32-zlib) [ 1707.449864] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1707.697543] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1709.375197] Key type lgssc registered [ 1710.785270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.499499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.526121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.118683] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1721.173341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1721.536647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.611656] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1739.162089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.248907] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1739.297475] Lustre: 57550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1739.543469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1744.634425] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1752.888617] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1759.187129] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1768.176595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1768.482374] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1768.789628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.518350] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1778.568576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1790.419566] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1798.847798] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1716h00m13s [ 1800.687704] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1808.946112] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1811.495734] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1811.641264] LustreError: 58816:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1831.838352] Lustre: lustre-MDT0000: Client 724c5616-dcbf-4198-8a45-d6b2a94a7dc9 (at 192.168.201.49@tcp) reconnecting [ 1851.727778] LustreError: 58816:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1854.278544] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1857.143716] Lustre: DEBUG MARKER: done [ 1865.700245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.703257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.725628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1865.737436] Lustre: Skipped 1 previous similar message [ 1870.819775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1870.839984] Lustre: Skipped 1 previous similar message [ 1871.286226] Lustre: server umount lustre-OST0000 complete [ 1875.938930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.940153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1875.963495] Lustre: Skipped 2 previous similar messages [ 1875.972742] Lustre: Skipped 1 previous similar message [ 1881.057087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1881.067931] Lustre: Skipped 3 previous similar messages [ 1881.587247] Lustre: server umount lustre-MDT0000 complete [ 1885.964586] LustreError: 57541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675301 with bad export cookie 11311764909145362225 [ 1885.972417] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1885.977283] LustreError: 57541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1886.325631] Lustre: server umount lustre-MDT0001 complete [ 1899.650343] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1903.686361] Key type lgssc unregistered [ 1904.152321] LNet: 61894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.163811] LNetError: 61894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.198268] LNet: Removed LNI 192.168.201.149@tcp [ 1905.625185] Key type .llcrypt unregistered [ 1905.630185] Key type ._llcrypt unregistered [ 1960.213144] Key type ._llcrypt registered [ 1960.215645] Key type .llcrypt registered [ 1960.320300] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1961.402019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1961.611294] alg: No test for adler32 (adler32-zlib) [ 1962.723925] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1962.962396] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1964.623304] Key type lgssc registered [ 1966.443667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1971.383307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1971.417853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.822600] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1972.879817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1973.146429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.345259] Lustre: Failing over lustre-MDT0000 [ 1974.772399] Lustre: server umount lustre-MDT0000 complete [ 1982.845261] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:03:16 (1789675396) [ 1990.335714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.856879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.465641] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2003.961253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.169848] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2004.213919] Lustre: 64689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2004.530277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2009.322980] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2017.018321] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2023.500137] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2030.889474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2031.060077] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2031.288956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.119417] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2037.455164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2046.231557] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2053.059101] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2063.329681] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.334181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2063.345838] Lustre: Skipped 2 previous similar messages [ 2063.355041] Lustre: Skipped 2 previous similar messages [ 2067.856548] Lustre: server umount lustre-MDT0000 complete [ 2072.346846] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675487 with bad export cookie 165850731398151421 [ 2072.357437] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.363076] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2072.920554] Lustre: server umount lustre-MDT0001 complete [ 2076.878331] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2081.778605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2088.927159] Lustre: 63611:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789675488/real 1789675488] req@00000000e2528f62 x1876610649204608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789675504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2088.982353] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.406817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.151451] Lustre: 63611:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789675492/real 1789675492] req@00000000aa66a998 x1876610649204736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789675508 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2097.251824] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x24d385c8ab9c5e0 [ 2097.277426] Lustre: MGC192.168.201.149@tcp: Connection restored to 0@lo (at 0@lo) [ 2097.595177] LustreError: 66540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.626213] LustreError: 66540:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2097.739421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.853139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2098.655126] Lustre: 63614:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789675497/real 1789675497] req@0000000011f9b5c8 x1876610649205120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789675513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2098.705204] Lustre: 63614:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2098.734566] LustreError: 66541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.764287] LustreError: 66540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.796138] LustreError: 66540:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2102.412871] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2103.844530] Lustre: 63612:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789675502/real 1789675502] req@000000009cce9efc x1876610649205248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789675518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2104.817487] LustreError: 68466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2104.841317] LustreError: 68466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2109.924907] LustreError: 66541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.973083] LustreError: 66541:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2112.874071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.468993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2118.591376] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2118.625675] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2128.265519] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2135.715650] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2144.226990] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.228336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.230343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2144.257409] Lustre: Skipped 2 previous similar messages [ 2144.293267] Lustre: Skipped 2 previous similar messages [ 2149.364367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.373415] Lustre: Skipped 2 previous similar messages [ 2150.010323] Lustre: server umount lustre-MDT0000 complete [ 2154.466493] LustreError: 69655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.499926] LustreError: 69655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2154.788941] LustreError: 67606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675569 with bad export cookie 165850731398153696 [ 2154.794237] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.798793] LustreError: 67606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2155.273236] Lustre: server umount lustre-MDT0001 complete [ 2164.177589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2174.945375] Lustre: 63614:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789675574/real 1789675574] req@000000008be5f4ff x1876610649233920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789675590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2174.971675] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.899550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.124477] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x24d385c8ab9cb43 [ 2179.152447] Lustre: MGC192.168.201.149@tcp: Connection restored to 0@lo (at 0@lo) [ 2179.183743] Lustre: Skipped 1 previous similar message [ 2179.701283] LustreError: 66541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.781631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.843102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2184.873235] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2185.891211] Lustre: 63611:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789675584/real 1789675584] req@000000007bc17f96 x1876610649234304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789675600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.925192] Lustre: 63611:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2185.981980] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2196.640806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.101518] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2202.566613] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2212.255335] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2218.910579] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2227.684283] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.698435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.711464] Lustre: Skipped 1 previous similar message [ 2227.727861] Lustre: Skipped 2 previous similar messages [ 2232.205341] Lustre: server umount lustre-MDT0000 complete [ 2232.801967] LustreError: 71152:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.810465] LustreError: 71152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2235.677693] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675650 with bad export cookie 165850731398155075 [ 2235.682320] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2235.692243] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2236.013821] Lustre: server umount lustre-MDT0001 complete [ 2242.491501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2246.333870] Lustre: DEBUG MARKER: files compared the same [ 2257.273486] Lustre: server umount lustre-OST0000 complete [ 2274.914703] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2278.172646] Key type lgssc unregistered [ 2278.574643] LNet: 74425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.593101] LNetError: 74425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.621856] LNet: Removed LNI 192.168.201.149@tcp [ 2279.774208] Key type .llcrypt unregistered [ 2279.776191] Key type ._llcrypt unregistered [ 2308.279435] Key type ._llcrypt registered [ 2308.281435] Key type .llcrypt registered [ 2308.438114] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2309.779781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.917311] alg: No test for adler32 (adler32-zlib) [ 2310.917451] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 2311.130789] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2312.783285] Key type lgssc registered [ 2313.876419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2321.821184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2321.842397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.238264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.702728] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug +malloc all [ 2333.674622] LustreError: 75839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.690641] LustreError: 75839:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2338.677068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.786842] LustreError: 75840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.134095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2344.944675] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug +malloc all [ 2352.318743] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2358.761883] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2367.559764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2367.892335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.955547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2374.966533] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug +malloc all [ 2383.288754] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2394.599688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2394.603917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.608250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2394.609678] Lustre: Skipped 1 previous similar message [ 2398.550406] Lustre: server umount lustre-OST0000 complete [ 2404.849035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.863404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.869017] Lustre: Skipped 1 previous similar message [ 2404.891555] Lustre: Skipped 2 previous similar messages [ 2409.284409] Lustre: server umount lustre-MDT0000 complete [ 2409.951988] LustreError: 75835:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.968596] LustreError: 75835:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2414.049675] LustreError: 75833:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.093641] LustreError: 75833:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2414.273836] LustreError: 75819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789675829 with bad export cookie 1243196258875029425 [ 2414.292516] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.303876] LustreError: 75819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2414.709731] Lustre: server umount lustre-MDT0001 complete [ 2429.278696] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2432.650318] Key type lgssc unregistered [ 2433.057490] LNet: 79372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.067183] LNetError: 79372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2433.084755] LNet: Removed LNI 192.168.201.149@tcp [ 2434.090839] Key type .llcrypt unregistered [ 2434.092876] Key type ._llcrypt unregistered [ 2459.002347] Key type ._llcrypt registered [ 2459.007347] Key type .llcrypt registered [ 2459.180238] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2460.481755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2460.688677] alg: No test for adler32 (adler32-zlib) [ 2461.881541] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 2462.201489] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2463.928683] Key type lgssc registered [ 2465.055257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2473.976905] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2474.009519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.695431] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2494.815939] LustreError: 80759:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.149@tcp: failed processing log, type 4: rc = -110 [ 2521.504469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.377620] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2535.994458] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2545.575869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.777537] Lustre: *** cfs_fail_loc=706, val=0*** [ 2546.263226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.357897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2551.275845] LustreError: 80784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.295278] LustreError: 80784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2552.169544] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2556.392639] LustreError: 80783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.421135] LustreError: 80783:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2561.516726] LustreError: 81845:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.537505] LustreError: 81845:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2562.229947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2562.716959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2568.795986] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2579.144699] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2586.366421] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2592.228242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.234292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.252813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.254697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.275364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.314240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.368804] Lustre: server umount lustre-OST0000 complete [ 2603.488947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2603.498188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.516080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2603.521373] Lustre: Skipped 1 previous similar message [ 2605.845444] Lustre: server umount lustre-MDT0000 complete [ 2609.121400] LustreError: 83577:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.148695] LustreError: 83577:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2609.964297] LustreError: 80765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789676025 with bad export cookie 15010439672773133806 [ 2609.989044] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2610.453440] Lustre: server umount lustre-MDT0001 complete [ 2622.860914] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2625.685942] Key type lgssc unregistered [ 2626.001772] LNet: 84375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.007206] LNetError: 84375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.021682] LNet: Removed LNI 192.168.201.149@tcp [ 2627.034918] Key type .llcrypt unregistered [ 2627.037775] Key type ._llcrypt unregistered [ 2652.863744] Key type ._llcrypt registered [ 2652.866589] Key type .llcrypt registered [ 2652.969559] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2653.894632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2653.932140] alg: No test for adler32 (adler32-zlib) [ 2655.069409] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 2655.415472] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2657.119980] Key type lgssc registered [ 2658.429646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2666.759779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2666.807047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.952848] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2681.347674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.724296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.570863] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2686.946925] LustreError: 86476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.968615] LustreError: 86476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2691.047375] LustreError: 86479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.086318] LustreError: 86479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2696.161841] LustreError: 86480:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.181399] LustreError: 86480:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2696.203159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.594653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.792820] LustreError: 86479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.817482] LustreError: 86479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2704.231709] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2706.917687] LustreError: 86475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.941048] LustreError: 86475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2715.023125] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2715.052216] Lustre: MGS: Client eb4ef5fd-30d0-4ba9-a2a3-b11f87e2eab1 (at 0@lo) reconnecting [ 2715.079743] LustreError: 86490:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2715.418264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.563866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2721.229510] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2741.220635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.227643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.232267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.245624] Lustre: Skipped 1 previous similar message [ 2742.934638] Lustre: server umount lustre-OST0000 complete [ 2751.457932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.468111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2751.496050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.517517] Lustre: Skipped 2 previous similar messages [ 2753.359131] Lustre: server umount lustre-MDT0000 complete [ 2756.583902] LustreError: 88021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.616434] LustreError: 88021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2760.547598] Lustre: server umount lustre-MDT0001 complete [ 2766.229613] Lustre: server umount MGS complete [ 2779.950049] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:16:34 (1789676194) [ 2812.315150] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 2825.732338] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2866.228484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.828727] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2881.419370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.765531] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2881.811525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2881.947825] Lustre: lustre-MDT0001: new disk, initializing [ 2882.068793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2882.105906] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2882.118673] Lustre: 93667:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2882.139189] LustreError: 93667:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2887.781609] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2899.541144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2899.882843] Lustre: lustre-OST0000: new disk, initializing [ 2899.895754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2899.903899] Lustre: 94432:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2900.051621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2907.248510] Lustre: 94693:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004f590049 x1876611374819840/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789676338 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2908.347114] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2919.280643] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2927.600763] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2927.876933] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2927.902208] Lustre: MGS: Client 0bc56c7b-df5f-4101-8ca9-9b62d795c4c7 (at 0@lo) reconnecting [ 2927.982387] Lustre: 95479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2927.999790] LustreError: 93653:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2928.011491] LustreError: 93653:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2928.031304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2928.133346] Lustre: lustre-MDT0000: new disk, initializing [ 2928.296294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.314506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2929.384645] Lustre: 95503:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000074e8b0e x1876611374827392/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789676360 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2933.754309] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2934.560916] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2938.483403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2938.511514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2938.654767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2963.443695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2963.456046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.471636] Lustre: Skipped 1 previous similar message [ 2963.478478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2966.391468] Lustre: server umount lustre-OST0000 complete [ 2974.176897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2974.183489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.186297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.195966] Lustre: Skipped 2 previous similar messages [ 2974.214701] Lustre: Skipped 2 previous similar messages [ 2976.934525] Lustre: server umount lustre-MDT0000 complete [ 2979.303344] LustreError: 93637:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.321869] LustreError: 93637:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2995.167576] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2995.440518] Lustre: server umount lustre-MDT0001 complete [ 2999.972334] Lustre: server umount MGS complete [ 3013.196979] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:20:27 (1789676427) [ 3032.682461] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3036.041361] Key type lgssc unregistered [ 3036.360234] LNet: 98006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3036.388879] LNetError: 98006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3036.415044] LNet: Removed LNI 192.168.201.149@tcp [ 3037.382608] Key type .llcrypt unregistered [ 3037.386102] Key type ._llcrypt unregistered [ 3055.778795] Key type ._llcrypt registered [ 3055.780717] Key type .llcrypt registered [ 3055.896857] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3057.051849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3057.129784] alg: No test for adler32 (adler32-zlib) [ 3058.297576] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 3058.565140] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3060.233157] Key type lgssc registered [ 3061.351760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3069.764973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3069.778472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.292427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.831768] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3076.611461] LustreError: 99222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.631693] LustreError: 99222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3081.697311] LustreError: 99223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.724248] LustreError: 99222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.983272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.335493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3089.349610] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3096.989759] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3103.216518] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3111.036951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3111.552273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.710116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3118.840599] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3127.348939] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3138.528784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3138.535583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.564151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.578762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.578899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.405497] Lustre: server umount lustre-OST0000 complete [ 3150.824236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.831406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3150.844694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.859186] Lustre: Skipped 2 previous similar messages [ 3154.625953] Lustre: server umount lustre-MDT0000 complete [ 3155.938666] LustreError: 99223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.965689] LustreError: 99223:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3159.463478] LustreError: 99956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789676574 with bad export cookie 18036895899947567541 [ 3159.476168] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3159.480910] LustreError: 99956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3159.780367] Lustre: server umount lustre-MDT0001 complete [ 3174.209628] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3177.604666] Key type lgssc unregistered [ 3177.916700] LNet: 102756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.928511] LNetError: 102756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3177.950590] LNet: Removed LNI 192.168.201.149@tcp [ 3179.002890] Key type .llcrypt unregistered [ 3179.004485] Key type ._llcrypt unregistered [ 3197.739569] Key type ._llcrypt registered [ 3197.741687] Key type .llcrypt registered [ 3197.857916] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3198.812320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3198.850363] alg: No test for adler32 (adler32-zlib) [ 3200.088925] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 3200.422325] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3202.135197] Key type lgssc registered [ 3203.399542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3209.890843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3209.897333] LustreError: 103837:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3209.901930] LustreError: 103840:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3209.914072] LustreError: 103837:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3209.928281] LustreError: 103837:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3209.941266] LustreError: 103837:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3209.942817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.738151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.962828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.245192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3225.347639] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3232.047084] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing lsmod [ 3234.971452] LustreError: 105042:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3234.994519] LustreError: 105045:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3235.004325] LustreError: 105042:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 3235.010474] LustreError: 105042:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3235.023347] LustreError: 105042:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3235.029405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3235.511864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.557967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3240.940303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3240.956284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.986293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.952909] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.957851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.966603] Lustre: Skipped 1 previous similar message [ 3241.976397] Lustre: Skipped 1 previous similar message [ 3244.532874] Lustre: server umount lustre-MDT0000 complete [ 3245.097116] LustreError: 103877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.119947] LustreError: 103877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3248.373579] LustreError: 103859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789676663 with bad export cookie 14256403632207143861 [ 3248.375367] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3248.383242] LustreError: 103859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3249.313227] LustreError: 104307:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000074e8b0e x1876611946271872/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3249.469047] Lustre: server umount lustre-MDT0001 complete [ 3263.762682] Lustre: server umount lustre-OST0000 complete [ 3272.887931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.280329] LustreError: 106110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.366406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.904179] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3278.824740] LustreError: 106111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.943172] LustreError: 106110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.360834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.917558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3293.126480] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3301.246280] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3307.169761] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3314.683672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.100916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.249953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3322.408321] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3331.150728] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3339.744596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3339.752454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.770632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.821114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3342.826803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.837356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.862205] Lustre: Skipped 1 previous similar message [ 3347.940233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3347.958018] Lustre: Skipped 1 previous similar message [ 3353.055415] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3353.202390] Lustre: server umount lustre-OST0000 complete [ 3358.181379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.185501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3358.198599] Lustre: Skipped 1 previous similar message [ 3358.221105] Lustre: Skipped 3 previous similar messages [ 3363.720452] Lustre: server umount lustre-MDT0000 complete [ 3368.135687] LustreError: 106092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789676783 with bad export cookie 14256403632207144792 [ 3368.162665] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3368.726428] Lustre: server umount lustre-MDT0001 complete [ 3381.298632] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3384.256688] Key type lgssc unregistered [ 3384.518560] LNet: 109644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3384.527213] LNetError: 109644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3384.548924] LNet: Removed LNI 192.168.201.149@tcp [ 3385.390553] Key type .llcrypt unregistered [ 3385.393208] Key type ._llcrypt unregistered [ 3411.176324] Key type ._llcrypt registered [ 3411.179600] Key type .llcrypt registered [ 3411.306335] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3412.640355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3412.655180] alg: No test for adler32 (adler32-zlib) [ 3413.848131] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 3414.218527] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3415.937268] Key type lgssc registered [ 3417.063189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3426.546604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3426.576511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3428.371411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.774923] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3438.567474] LustreError: 111055:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.586798] LustreError: 111055:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3443.683429] LustreError: 111056:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.702871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.044454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3449.415401] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3457.705185] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3463.763844] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3471.666161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.095096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.149190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3479.557608] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3489.039487] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3495.519395] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3499.841726] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3505.123152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.126612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3505.150684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3505.176912] Lustre: Skipped 1 previous similar message [ 3510.253777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3510.259020] Lustre: Skipped 1 previous similar message [ 3511.020041] Lustre: server umount lustre-OST0000 complete [ 3515.363100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.380344] Lustre: Skipped 1 previous similar message [ 3515.387945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3520.482910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3520.489086] Lustre: Skipped 2 previous similar messages [ 3522.094850] Lustre: server umount lustre-MDT0000 complete [ 3525.612087] LustreError: 111055:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.649255] LustreError: 111055:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3526.880914] LustreError: 111037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789676941 with bad export cookie 3122058503105080594 [ 3526.899236] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3527.823769] Lustre: server umount lustre-MDT0001 complete [ 3544.543219] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3548.292612] Key type lgssc unregistered [ 3548.567913] LNet: 114696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3548.575908] LNetError: 114696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3549.609847] LNet: Removed LNI 192.168.201.149@tcp [ 3550.586335] Key type .llcrypt unregistered [ 3550.591385] Key type ._llcrypt unregistered [ 3569.005441] Key type ._llcrypt registered [ 3569.008869] Key type .llcrypt registered [ 3569.131430] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3570.111678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3570.155160] alg: No test for adler32 (adler32-zlib) [ 3571.828191] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 3572.379767] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3574.215292] Key type lgssc registered [ 3575.764093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3585.761730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3585.780030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.321498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.411810] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3597.803607] LustreError: 115919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.824690] LustreError: 115919:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3601.355343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.797875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3606.903037] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3615.896533] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3621.905779] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3628.217733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.477512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.592031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3634.949259] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3642.704562] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3657.196546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.201191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3657.202845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3657.203686] Lustre: Skipped 1 previous similar message [ 3662.331943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3662.348820] Lustre: Skipped 1 previous similar message [ 3662.827283] Lustre: server umount lustre-OST0000 complete [ 3667.427340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.433209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3667.446795] Lustre: Skipped 1 previous similar message [ 3667.465190] Lustre: Skipped 1 previous similar message [ 3672.548667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3672.557925] Lustre: Skipped 1 previous similar message [ 3672.988477] Lustre: server umount lustre-MDT0000 complete [ 3676.910584] LustreError: 117763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789677091 with bad export cookie 15503272570855464872 [ 3676.934609] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3676.942441] LustreError: 117763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3677.326154] Lustre: server umount lustre-MDT0001 complete [ 3689.000848] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3692.015284] Key type lgssc unregistered [ 3692.447293] LNet: 119480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3692.459407] LNetError: 119480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3693.495081] LNet: Removed LNI 192.168.201.149@tcp [ 3694.621901] Key type .llcrypt unregistered [ 3694.624224] Key type ._llcrypt unregistered [ 3721.898939] Key type ._llcrypt registered [ 3721.901286] Key type .llcrypt registered [ 3722.005727] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3723.735632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3723.777547] alg: No test for adler32 (adler32-zlib) [ 3724.789431] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 3725.146820] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3726.880754] Key type lgssc registered [ 3727.881833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3734.851620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3734.868412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3736.333944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.064757] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3746.786096] LustreError: 120893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3746.796215] LustreError: 120893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3749.590090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.021845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3754.939569] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3764.038712] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3771.020359] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3777.842584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3778.139539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.250195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3785.106573] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3793.422489] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3800.383623] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3803.991813] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3804.003143] Lustre: Skipped 2 previous similar messages [ 3808.324716] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3808.330843] Lustre: Skipped 2 previous similar messages [ 3808.891656] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3808.925208] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3819.334647] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.49@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3819.351906] Lustre: Skipped 2 previous similar messages [ 3825.323946] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.49@tcp 192.168.0.[2,10]@tcp [ 3829.044445] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.49/24@tcp in log lustre-MDT0000 [ 3829.049942] Lustre: Skipped 2 previous similar messages [ 3837.516118] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.49/24@tcp [ 3837.523872] Lustre: Skipped 1 previous similar message [ 3843.040173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3843.058614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.065227] LustreError: Skipped 3 previous similar messages [ 3843.084596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.683245] Lustre: server umount lustre-OST0000 complete [ 3851.236100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.245387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3851.266851] Lustre: Skipped 1 previous similar message [ 3851.280028] Lustre: Skipped 1 previous similar message [ 3854.906188] Lustre: server umount lustre-MDT0000 complete [ 3856.358376] LustreError: 120894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3856.380867] LustreError: 120894:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3858.827697] LustreError: 120873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789677273 with bad export cookie 17576784607336176730 [ 3858.829703] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3858.840024] LustreError: 120873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3868.413028] Lustre: server umount lustre-MDT0001 complete [ 3884.227625] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3887.655939] Key type lgssc unregistered [ 3888.181554] LNet: 125780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3888.200722] LNetError: 125780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3889.260782] LNet: Removed LNI 192.168.201.149@tcp [ 3890.430892] Key type .llcrypt unregistered [ 3890.433383] Key type ._llcrypt unregistered [ 3925.036463] Key type ._llcrypt registered [ 3925.043311] Key type .llcrypt registered [ 3925.272706] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3926.592313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3926.834630] alg: No test for adler32 (adler32-zlib) [ 3927.880451] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 3928.155132] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3929.863321] Key type lgssc registered [ 3931.119295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3938.063415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3938.116169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3939.513411] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3939.541397] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.49/24@tcp [ 3939.543493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3943.974671] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3949.030894] LustreError: 127339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.056481] LustreError: 127339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3952.984097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3953.273070] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3953.323433] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.49/24@tcp [ 3953.359210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3958.841457] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3967.997618] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3974.640966] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3982.493495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3982.953590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3985.052105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3988.744401] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3997.542607] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4007.909179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4007.919587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.950559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.956680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.965903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.970593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.009688] Lustre: Skipped 1 previous similar message [ 4014.315627] Lustre: server umount lustre-OST0000 complete [ 4020.192823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.195742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.220995] Lustre: Skipped 1 previous similar message [ 4020.236878] Lustre: Skipped 1 previous similar message [ 4025.296404] Lustre: server umount lustre-MDT0000 complete [ 4025.317330] LustreError: 127339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4025.335598] LustreError: 127339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4029.314034] LustreError: 130177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789677444 with bad export cookie 10151250199617312459 [ 4029.316498] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4029.343169] LustreError: 130177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4029.623544] Lustre: server umount lustre-MDT0001 complete [ 4042.376763] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4045.324821] Key type lgssc unregistered [ 4045.648795] LNet: 130929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4045.654372] LNetError: 130929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4046.709976] LNet: Removed LNI 192.168.201.149@tcp [ 4047.848145] Key type .llcrypt unregistered [ 4047.850122] Key type ._llcrypt unregistered [ 4086.689917] Key type ._llcrypt registered [ 4086.692105] Key type .llcrypt registered [ 4086.786754] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 4105.559974] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4106.964571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4107.015376] alg: No test for adler32 (adler32-zlib) [ 4108.007326] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 4108.270693] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4109.984177] Key type lgssc registered [ 4111.212168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4147.390895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4147.425431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.560816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4148.584238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4148.633980] Lustre: lustre-MDT0000: new disk, initializing [ 4148.672116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.692480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4154.189749] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4166.585123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.758882] Lustre: 134329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4166.843984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4166.849545] Lustre: Skipped 1 previous similar message [ 4166.991418] Lustre: lustre-MDT0001: new disk, initializing [ 4167.158654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.228256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4167.245256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4171.936781] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4182.791729] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4189.333660] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4197.367713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4197.739365] Lustre: lustre-OST0000: new disk, initializing [ 4197.747177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4197.753482] Lustre: 136295:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4197.867945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4204.635595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4204.651775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4204.764199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4205.234970] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4216.890950] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4221.713208] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4221.972070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4226.246608] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4226.410230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4230.623782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4230.625575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.639512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4230.641623] Lustre: Skipped 1 previous similar message [ 4230.663893] Lustre: Skipped 1 previous similar message [ 4234.728086] Lustre: server umount lustre-OST0000 complete [ 4238.818504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4238.826575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.836704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4240.868214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4240.870687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.991922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4246.016353] Lustre: Skipped 2 previous similar messages [ 4250.089163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4250.100318] Lustre: Skipped 1 previous similar message [ 4253.151480] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4253.423839] Lustre: server umount lustre-MDT0000 complete [ 4258.684123] LustreError: 134321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789677673 with bad export cookie 676331478113559103 [ 4258.705784] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4258.714041] LustreError: 134321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4259.377176] Lustre: server umount lustre-MDT0001 complete [ 4271.788199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.409453] LustreError: 138520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4272.450759] LustreError: 138520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4272.687437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.729096] LustreError: 138521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.958727] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4282.852716] LustreError: 138520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4287.972298] LustreError: 138521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.732895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.201299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4293.987771] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4302.460970] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4309.638828] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4317.176064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4317.480420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.780110] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4333.469384] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4338.780278] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4342.403893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4348.173440] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4348.627240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4359.345539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4359.567772] Lustre: lustre-OST0001: new disk, initializing [ 4359.573393] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4359.582946] Lustre: 141913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4359.718221] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4366.608316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4366.619623] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4366.753554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4367.291366] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4377.315977] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4377.584521] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4382.657153] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4382.988384] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4389.757928] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98e5c618b800.ost_server_uuid 50 [ 4391.782659] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98e5c618b800.ost_server_uuid in IDLE state after 0 sec [ 4401.552304] Lustre: server umount lustre-OST0001 complete [ 4402.659564] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4402.670762] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.681202] LustreError: Skipped 1 previous similar message [ 4402.681894] LustreError: 140342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.713535] Lustre: Skipped 1 previous similar message [ 4406.760971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4406.764822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.790157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4411.759540] Lustre: server umount lustre-OST0000 complete [ 4416.992321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4416.995835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.010779] Lustre: Skipped 1 previous similar message [ 4422.023988] Lustre: server umount lustre-MDT0000 complete [ 4423.151177] LustreError: 138520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.185195] LustreError: 138520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4426.314111] LustreError: 138502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789677841 with bad export cookie 676331478113559978 [ 4426.319166] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4426.324501] LustreError: 138502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4426.727874] Lustre: server umount lustre-MDT0001 complete [ 4450.232629] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4453.188564] Key type lgssc unregistered [ 4453.554720] LNet: 144916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4453.573109] LNetError: 144916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4453.597854] LNet: Removed LNI 192.168.201.149@tcp [ 4454.429767] Key type .llcrypt unregistered [ 4454.432366] Key type ._llcrypt unregistered [ 4495.447266] Key type ._llcrypt registered [ 4495.448821] Key type .llcrypt registered [ 4495.571629] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 4510.746754] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4512.657487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4512.741870] alg: No test for adler32 (adler32-zlib) [ 4513.878971] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 4514.250751] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4515.943159] Key type lgssc registered [ 4516.939248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4556.595622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4556.643431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.977783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4558.009097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4558.127308] Lustre: lustre-MDT0000: new disk, initializing [ 4558.252338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.279827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4562.791234] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4574.692974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4574.893713] Lustre: 149074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4575.002727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4575.010781] Lustre: Skipped 1 previous similar message [ 4575.114269] Lustre: lustre-MDT0001: new disk, initializing [ 4575.263959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4575.318055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4575.343652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4580.647785] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4590.750545] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4596.705086] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4604.316091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4604.697959] Lustre: lustre-OST0000: new disk, initializing [ 4604.702093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4604.714372] Lustre: 151040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4604.801827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4610.172175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4610.189367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4610.339871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4611.780209] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4621.273873] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4624.806345] Lustre: Failing over lustre-OST0000 [ 4624.918969] Lustre: server umount lustre-OST0000 complete [ 4641.247196] Lustre: 147296:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789678040/real 1789678040] req@00000000697b49e0 x1876613324113664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789678056 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4641.293941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4642.783372] Lustre: 147298:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789678040/real 1789678040] req@00000000ef06fb1e x1876613324112896/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789678056 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4642.809263] Lustre: 147298:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4643.060315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4643.428808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.452771] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4644.449525] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789678043/real 1789678043] req@00000000a389c526 x1876613324114176/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789678059 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4644.485502] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4647.456399] Lustre: 147297:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789678046/real 1789678046] req@00000000906148c2 x1876613324115072/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789678062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4647.498719] Lustre: 147297:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4648.313899] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4649.990737] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4651.666064] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4651.675527] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4652.703104] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789678051/real 1789678051] req@00000000b25bd070 x1876613324116224/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789678067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4653.521474] Lustre: Failing over lustre-MDT0000 [ 4655.806397] Lustre: server umount lustre-MDT0000 complete [ 4657.128403] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.133720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4657.135978] LustreError: 149080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.135990] LustreError: 149080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4657.149121] Lustre: Skipped 3 previous similar messages [ 4658.581470] LustreError: 149080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.595264] LustreError: 149080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4662.240756] LustreError: 149081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.263578] LustreError: 149081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4667.366751] LustreError: 149084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.388631] LustreError: 149084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4672.489831] LustreError: 149085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4672.521762] LustreError: 149085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4673.505904] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789678072/real 1789678072] req@00000000ab95baf4 x1876613324126592/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789678088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4673.544264] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4673.553806] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4675.238591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.748376] LustreError: 147294:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000048c80182 x1876613324132480/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4683.748897] LustreError: 149084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4683.768647] LustreError: 149084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 4683.960890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.014224] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4684.169414] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4688.567886] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4689.385993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4689.394689] Lustre: Skipped 1 previous similar message [ 4689.404335] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4689.440590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4698.090668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4698.101987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.128320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4699.641656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4699.655056] Lustre: Skipped 1 previous similar message [ 4704.748950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4704.753695] Lustre: Skipped 1 previous similar message [ 4709.869973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4709.883935] Lustre: Skipped 1 previous similar message [ 4711.391436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4711.553631] Lustre: server umount lustre-OST0000 complete [ 4720.096482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.098127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4720.119611] Lustre: Skipped 1 previous similar message [ 4720.142792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4720.155648] Lustre: Skipped 1 previous similar message [ 4722.066789] Lustre: server umount lustre-MDT0000 complete [ 4725.218951] LustreError: 149084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4725.240797] LustreError: 149084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4726.523674] LustreError: 149063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789678141 with bad export cookie 17407218567966228564 [ 4726.534527] LustreError: 149063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4726.535759] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4726.835961] Lustre: server umount lustre-MDT0001 complete [ 4737.617608] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4740.146872] Key type lgssc unregistered [ 4740.427787] LNet: 154701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4740.444791] LNetError: 154701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4740.463842] LNet: Removed LNI 192.168.201.149@tcp [ 4741.342784] Key type .llcrypt unregistered [ 4741.346758] Key type ._llcrypt unregistered [ 4779.943764] Key type ._llcrypt registered [ 4779.948710] Key type .llcrypt registered [ 4780.154851] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4781.213969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4781.246472] alg: No test for adler32 (adler32-zlib) [ 4782.321760] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 4782.525578] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4784.159250] Key type lgssc registered [ 4785.742720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4794.737701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4794.746666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.446770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4802.331444] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4806.630239] LustreError: 156220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4806.661796] LustreError: 156220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4811.747426] LustreError: 156219:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4814.590631] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4816.870256] LustreError: 156220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.534163] LustreError: 156248:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4819.550922] LustreError: 156248:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4819.564581] LustreError: 156248:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 4819.990900] Lustre: server umount lustre-MDT0000 complete [ 4838.290934] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 4854.466130] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4891.548926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4891.807209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4891.835332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4891.966507] Lustre: lustre-MDT0000: new disk, initializing [ 4892.044819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4892.058782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4896.506088] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4908.924198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.091396] Lustre: 159578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4909.158852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4909.166942] Lustre: Skipped 1 previous similar message [ 4909.244777] Lustre: lustre-MDT0001: new disk, initializing [ 4909.320870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4909.373250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4909.384933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4914.135897] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4924.972422] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4931.544337] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4938.918369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4939.247643] Lustre: lustre-OST0000: new disk, initializing [ 4939.251684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4939.259278] Lustre: 161549:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4939.342293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.710617] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4948.026457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4948.047712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4948.111528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4954.952851] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4959.333705] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4960.997496] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:52:55 (1789678375) [ 5221.860653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5221.868077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.884770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5224.931383] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.940796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5224.945061] Lustre: Skipped 1 previous similar message [ 5224.958826] Lustre: Skipped 2 previous similar messages [ 5226.806390] Lustre: server umount lustre-MDT0000 complete [ 5230.052079] LustreError: 160803:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.694609] LustreError: 161548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789678645 with bad export cookie 971749152172571073 [ 5230.696931] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5230.708033] LustreError: 161548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5235.168501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5235.169563] LustreError: 159584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.181633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5235.188574] LustreError: 159584:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5237.039860] Lustre: server umount lustre-MDT0001 complete [ 5247.278204] Lustre: server umount lustre-OST0000 complete [ 5256.593652] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 5265.316218] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5314.064348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5314.441978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5314.465974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5314.567141] Lustre: lustre-MDT0000: new disk, initializing [ 5314.637783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5314.664189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5320.567548] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5331.909166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5332.060696] Lustre: 166216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5332.161869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5332.164833] Lustre: Skipped 1 previous similar message [ 5332.329942] Lustre: lustre-MDT0001: new disk, initializing [ 5332.482653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5332.535231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5332.549143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5338.266037] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5349.013571] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5355.504545] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5363.429619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5363.790178] Lustre: lustre-OST0000: new disk, initializing [ 5363.794896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5363.809380] Lustre: 168184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5364.013730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5370.587779] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5372.995564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5373.011090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5373.172940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5381.368454] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5387.135221] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5387.401626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5393.845851] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5394.224388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5399.019090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5399.033116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.040462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5399.061635] Lustre: Skipped 1 previous similar message [ 5399.073499] Lustre: Skipped 1 previous similar message [ 5404.130109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5404.142537] Lustre: Skipped 1 previous similar message [ 5411.295193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5411.699938] Lustre: server umount lustre-OST0000 complete [ 5419.488768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.497403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5419.507148] Lustre: Skipped 1 previous similar message [ 5419.523978] Lustre: Skipped 3 previous similar messages [ 5421.870633] Lustre: server umount lustre-MDT0000 complete [ 5424.614489] LustreError: 166223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5424.634501] LustreError: 166223:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5426.148701] LustreError: 167010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789678841 with bad export cookie 971749152172729903 [ 5426.157978] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5426.161628] LustreError: 167010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5426.546252] Lustre: server umount lustre-MDT0001 complete [ 5436.861089] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:00:50 (1789678850) [ 5456.913464] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 5467.949975] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5510.534148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.734921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5510.781983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5510.941431] Lustre: lustre-MDT0000: new disk, initializing [ 5511.055598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.074319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5516.044568] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5527.857110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5528.038415] Lustre: 172956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5528.399000] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5534.025341] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5543.950941] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5549.368394] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5555.949569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5556.354903] Lustre: lustre-OST0000: new disk, initializing [ 5556.360881] Lustre: Skipped 1 previous similar message [ 5556.371102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5556.383587] Lustre: Skipped 2 previous similar messages [ 5556.390150] Lustre: 174934:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5561.955383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5561.968541] Lustre: Skipped 1 previous similar message [ 5561.977983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5562.152821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5563.558651] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5575.242222] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5587.937881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5587.948899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.958117] LustreError: Skipped 2 previous similar messages [ 5587.958810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.983120] Lustre: Skipped 1 previous similar message [ 5591.147250] Lustre: server umount lustre-OST0000 complete [ 5601.854359] Lustre: server umount lustre-MDT0000 complete [ 5603.299457] LustreError: 172963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5603.321544] LustreError: 172963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5606.005689] LustreError: 173766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789679021 with bad export cookie 971749152172730785 [ 5606.009190] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.024090] LustreError: 173766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5606.520431] Lustre: server umount lustre-MDT0001 complete [ 5616.176892] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:03:50 (1789679030) [ 5635.108134] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 5646.619873] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5683.915340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5684.268172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5684.336849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5684.416669] Lustre: lustre-MDT0000: new disk, initializing [ 5684.492743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5684.503445] Lustre: Skipped 2 previous similar messages [ 5684.527644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5688.900510] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5699.241315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.423965] Lustre: 179259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5699.755186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5704.862473] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5714.437686] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5719.810697] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5726.050043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5726.341451] Lustre: 181234:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5732.711827] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5733.463040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5733.583817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5742.025743] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5749.223099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.229512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5749.237801] Lustre: Skipped 3 previous similar messages [ 5749.238615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5749.238621] Lustre: Skipped 3 previous similar messages [ 5749.288481] LustreError: Skipped 1 previous similar message [ 5755.390431] Lustre: server umount lustre-OST0000 complete [ 5765.746661] Lustre: server umount lustre-MDT0000 complete [ 5769.103540] LustreError: 179252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789679184 with bad export cookie 971749152172731870 [ 5769.109664] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5769.118347] LustreError: 179252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5769.658792] Lustre: server umount lustre-MDT0001 complete [ 5781.919575] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 5784.701296] Key type lgssc unregistered [ 5785.072465] LNet: 183090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5785.083520] LNetError: 183090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5785.096543] LNet: Removed LNI 192.168.201.149@tcp [ 5785.916535] Key type .llcrypt unregistered [ 5785.919752] Key type ._llcrypt unregistered [ 5813.044757] Key type ._llcrypt registered [ 5813.048802] Key type .llcrypt registered [ 5813.232107] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5815.537745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5815.649368] alg: No test for adler32 (adler32-zlib) [ 5817.008512] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 5817.431200] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 5819.512461] Key type lgssc registered [ 5821.300985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5833.453428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5833.474577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.170133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.279777] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5845.476071] LustreError: 184502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.504273] LustreError: 184502:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5850.602669] LustreError: 184503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.512636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.753160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5857.106974] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5868.168700] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5876.915951] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5887.228612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5887.670764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5895.952308] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5905.126299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5905.819066] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5925.862487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.874020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5925.889727] Lustre: Skipped 1 previous similar message [ 5925.892772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5925.896990] Lustre: Skipped 1 previous similar message [ 5930.989885] LustreError: 186326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.144409] Lustre: server umount lustre-OST0000 complete [ 5938.658167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5938.667986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.687284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5941.227287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5941.230122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.433930] Lustre: server umount lustre-MDT0000 complete [ 5946.371332] LustreError: 184498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.392077] LustreError: 184498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5947.988678] LustreError: 184484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789679363 with bad export cookie 3747195936825622752 [ 5947.993183] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5948.008326] LustreError: 184484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5948.478533] Lustre: server umount lustre-MDT0001 complete [ 5963.779786] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 5967.232582] Key type lgssc unregistered [ 5967.468687] LNet: 188061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5967.477333] LNetError: 188061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5967.505852] LNet: Removed LNI 192.168.201.149@tcp [ 5968.791822] Key type .llcrypt unregistered [ 5968.797230] Key type ._llcrypt unregistered [ 5998.228384] Key type ._llcrypt registered [ 5998.231264] Key type .llcrypt registered [ 5998.337107] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6000.078122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6000.176351] alg: No test for adler32 (adler32-zlib) [ 6001.239381] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 6001.470259] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 6003.208385] Key type lgssc registered [ 6004.725152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6014.695143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6014.731165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6016.393714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.974296] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6026.729094] LustreError: 189476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.761069] LustreError: 189476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6030.044188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6030.546068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6035.042307] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6043.663399] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6051.102215] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6059.952502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6060.697385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6061.758232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6070.349463] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6079.687754] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6088.089917] Lustre: lustre-OST0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 6090.735305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.736822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.758891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.766452] Lustre: Skipped 1 previous similar message [ 6093.180936] Lustre: lustre-OST0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 6095.847525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6095.857130] Lustre: Skipped 1 previous similar message [ 6100.974245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6100.986514] Lustre: Skipped 2 previous similar messages [ 6101.984276] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6102.154470] Lustre: server umount lustre-OST0000 complete [ 6112.025984] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff98e5c8d4f800.ost_server_uuid 50 [ 6114.538744] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6129.447664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.449521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6129.467083] Lustre: Skipped 2 previous similar messages [ 6129.500959] Lustre: Skipped 1 previous similar message [ 6135.961165] Lustre: server umount lustre-MDT0000 complete [ 6136.801128] LustreError: 189476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.822160] LustreError: 189476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6140.293417] LustreError: 190210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789679555 with bad export cookie 14557242778587211801 [ 6140.299249] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6140.311546] LustreError: 190210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6140.944430] Lustre: server umount lustre-MDT0001 complete [ 6151.856461] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:12:45 (1789679565) [ 6158.861721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.393515] LustreError: 193394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.464142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.462731] LustreError: 193395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6165.283308] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6169.578233] LustreError: 193394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.842638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.348698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6182.913574] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6192.538615] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6202.421068] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6212.135722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6212.638088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.420688] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6228.054304] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6229.064601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6236.427769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6236.620678] Lustre: lustre-OST0001: new disk, initializing [ 6236.625896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6236.631428] Lustre: 196238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6236.752307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6243.547270] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6245.505142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6245.517494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6245.680089] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6253.227109] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6256.097216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.115207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6256.116959] Lustre: Skipped 1 previous similar message [ 6256.129096] Lustre: Skipped 3 previous similar messages [ 6262.232256] Lustre: server umount lustre-OST0000 complete [ 6264.196371] LustreError: 195219:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6264.231299] LustreError: 195219:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6268.889309] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6269.057698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6274.071980] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6274.262639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6281.699521] LustreError: 195217:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6281.724641] LustreError: 195217:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6291.944749] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.957284] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6291.963590] Lustre: Skipped 1 previous similar message [ 6291.964695] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6291.964704] Lustre: Skipped 3 previous similar messages [ 6292.003208] LustreError: Skipped 1 previous similar message [ 6295.220375] Lustre: server umount lustre-OST0001 complete [ 6300.128921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6300.141075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6305.802416] Lustre: server umount lustre-MDT0000 complete [ 6313.316618] LustreError: 193376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789679728 with bad export cookie 14557242778587212872 [ 6313.353691] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6314.302467] Lustre: server umount lustre-MDT0001 complete [ 6349.167882] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:16:03 (1789679763) [ 6357.719692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6358.011111] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6358.037415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6358.231520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6363.159488] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6373.459808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.631843] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6373.700224] Lustre: 199678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6378.856854] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6386.535548] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6393.562366] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6400.616669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6400.952598] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6408.214253] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6409.473533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6416.185540] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6424.205975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6424.385440] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6424.682841] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6424.692036] Lustre: Skipped 2 previous similar messages [ 6431.419082] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6432.391571] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6441.622601] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6445.536546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6445.543519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.545938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6445.553118] Lustre: Skipped 2 previous similar messages [ 6445.555470] Lustre: Skipped 4 previous similar messages [ 6450.887289] Lustre: server umount lustre-OST0000 complete [ 6452.197814] LustreError: 201532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.222751] LustreError: 201532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6467.040785] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6467.051304] LustreError: Skipped 1 previous similar message [ 6467.064680] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6473.305451] Lustre: server umount lustre-OST0001 complete [ 6481.376745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6481.391193] LustreError: Skipped 1 previous similar message [ 6484.595738] Lustre: server umount lustre-MDT0000 complete [ 6489.069436] LustreError: 199671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789679904 with bad export cookie 14557242778587214279 [ 6489.071637] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6489.083366] LustreError: 199671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6489.577854] Lustre: server umount lustre-MDT0001 complete [ 6518.710416] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:18:52 (1789679932) [ 6535.766399] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 6547.580322] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6584.936185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.216459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6585.245981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6585.341839] Lustre: lustre-MDT0000: new disk, initializing [ 6585.406175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6585.434200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6590.765837] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6603.260392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.449974] Lustre: 207779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6603.543168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6603.556068] Lustre: Skipped 1 previous similar message [ 6603.738536] Lustre: lustre-MDT0001: new disk, initializing [ 6603.900503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6603.917312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6609.881871] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6621.050876] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6627.667821] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6635.474282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6635.907969] Lustre: lustre-OST0000: new disk, initializing [ 6635.916417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6635.937748] Lustre: 209748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6642.243054] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6642.326884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6642.350589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6642.541169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6654.685815] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6662.270133] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6662.694662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6668.559221] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6669.051239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6673.390086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6673.403185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.414089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6673.425071] Lustre: Skipped 4 previous similar messages [ 6673.474601] Lustre: Skipped 12 previous similar messages [ 6686.175249] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6686.421513] Lustre: server umount lustre-OST0000 complete [ 6697.859767] Lustre: server umount lustre-MDT0000 complete [ 6698.977947] LustreError: 207786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6698.988819] LustreError: 207786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 6702.210612] LustreError: 208574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680117 with bad export cookie 14557242778587216064 [ 6702.219924] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6702.235865] LustreError: 208574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6702.544502] Lustre: server umount lustre-MDT0001 complete [ 6712.667623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.219275] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6726.783032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.633967] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6741.985491] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6749.171633] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6758.282885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6766.710901] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6775.499780] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6780.556851] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6780.770985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6785.869716] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6786.190770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6790.625370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6790.648266] LustreError: Skipped 1 previous similar message [ 6790.652985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6790.679254] Lustre: Skipped 3 previous similar messages [ 6794.686708] Lustre: server umount lustre-OST0000 complete [ 6801.915270] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6807.655873] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6839.712962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6847.570265] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6855.783108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6855.802853] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6855.805672] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6855.840894] Lustre: Skipped 1 previous similar message [ 6857.023332] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6859.584180] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6866.405552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6872.189185] Lustre: server umount lustre-OST0000 complete [ 6883.149987] Lustre: server umount lustre-MDT0000 complete [ 6886.886886] LustreError: 211967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.934252] LustreError: 211967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6888.094715] LustreError: 212703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680303 with bad export cookie 14557242778587216946 [ 6888.101900] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6888.112712] LustreError: 212703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6888.508097] Lustre: server umount lustre-MDT0001 complete [ 6902.739765] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:25:16 (1789680316) [ 6911.578033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.210927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6912.219130] Lustre: Skipped 6 previous similar messages [ 6917.957735] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6928.022364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.114180] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6943.536966] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6952.092628] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6961.907399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.052446] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6979.045539] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6985.715837] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6986.130341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6991.463641] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6991.608523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6999.305115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6999.535880] Lustre: lustre-OST0001: new disk, initializing [ 6999.540368] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6999.548557] Lustre: 221313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7006.608101] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7007.388386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7007.404197] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7007.525481] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7018.106218] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7023.552847] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7023.761224] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7029.144489] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7029.340654] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7033.321895] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7033.322906] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7033.331824] LustreError: Skipped 1 previous similar message [ 7033.336598] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7033.336606] Lustre: Skipped 16 previous similar messages [ 7033.387611] Lustre: Skipped 5 previous similar messages [ 7037.622505] Lustre: server umount lustre-OST0001 complete [ 7046.224390] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7046.529495] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7051.509106] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7051.848948] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7082.826117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.630111] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7088.656273] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7088.661257] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7088.692241] Lustre: Skipped 1 previous similar message [ 7090.791820] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7099.528405] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7102.090599] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7110.551376] Lustre: server umount lustre-OST0001 complete [ 7135.689905] Lustre: server umount lustre-OST0000 complete [ 7146.833503] Lustre: server umount lustre-MDT0000 complete [ 7150.048475] LustreError: 222153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7150.086801] LustreError: 222153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 39 previous similar messages [ 7152.037703] LustreError: 217925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680567 with bad export cookie 14557242778587217912 [ 7152.042931] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7152.063488] LustreError: 217925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7158.662834] Lustre: server umount lustre-MDT0001 complete [ 7186.009874] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:30:00 (1789680600) [ 7193.361544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.659462] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7193.689482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7198.716736] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7208.796890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.932755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7208.960697] Lustre: 226559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7213.690410] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7222.462634] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7229.610572] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7237.735183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7238.026780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7245.927512] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7255.382086] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7266.385983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7266.531036] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7274.234165] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7282.922470] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7284.994991] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7290.533204] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7290.729396] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7295.340595] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7295.531598] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7301.846898] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98e5c6563800.ost_server_uuid 50 [ 7303.401663] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98e5c6563800.ost_server_uuid in IDLE state after 0 sec [ 7304.249645] Lustre: Permanently deactivating lustre-OST0001 [ 7304.268598] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7308.311651] Lustre: Permanently reactivating lustre-OST0001 [ 7308.341209] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7308.345561] Lustre: Skipped 2 previous similar messages [ 7315.961730] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7315.963193] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.968625] LustreError: Skipped 5 previous similar messages [ 7315.984172] Lustre: Skipped 6 previous similar messages [ 7318.698652] Lustre: server umount lustre-OST0001 complete [ 7329.401747] Lustre: server umount lustre-OST0000 complete [ 7340.981588] Lustre: server umount lustre-MDT0000 complete [ 7346.191480] LustreError: 226554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680761 with bad export cookie 14557242778587219200 [ 7346.193282] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.199104] LustreError: 226554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7346.741729] Lustre: server umount lustre-MDT0001 complete [ 7375.216069] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:33:09 (1789680789) [ 7384.578485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.057249] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7385.104413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7385.112726] Lustre: Skipped 2 previous similar messages [ 7390.337559] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7398.519876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.660160] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7398.693971] Lustre: 233260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7404.884199] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7413.913739] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7420.809463] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7427.802780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7428.022883] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7428.037685] Lustre: Permanently deactivating lustre-OST0000 [ 7428.059895] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7428.346931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.361536] Lustre: Skipped 10 previous similar messages [ 7435.342660] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7435.451748] LustreError: 188785:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cf52259b x1876614883811712/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7436.258379] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7445.311202] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7454.329974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.426361] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7461.211692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7462.903328] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7473.793211] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7476.204658] Lustre: Permanently reactivating lustre-OST0000 [ 7476.255230] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7476.268766] Lustre: Skipped 2 previous similar messages [ 7482.444742] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7482.515871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7499.512559] Lustre: server umount lustre-OST0001 complete [ 7509.632836] Lustre: server umount lustre-OST0000 complete [ 7519.996124] Lustre: server umount lustre-MDT0000 complete [ 7523.821789] LustreError: 233253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789680938 with bad export cookie 14557242778587220873 [ 7523.829352] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7523.835429] LustreError: 233253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7524.311772] Lustre: server umount lustre-MDT0001 complete [ 7534.817845] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7537.387673] Key type lgssc unregistered [ 7537.692476] LNet: 238138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7537.700904] LNetError: 238138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7537.721431] LNet: Removed LNI 192.168.201.149@tcp [ 7538.511863] Key type .llcrypt unregistered [ 7538.514212] Key type ._llcrypt unregistered [ 7559.471958] Key type ._llcrypt registered [ 7559.475137] Key type .llcrypt registered [ 7559.555096] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7560.751438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7560.796588] alg: No test for adler32 (adler32-zlib) [ 7561.888426] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 7562.154115] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7563.864316] Key type lgssc registered [ 7564.836574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7573.746519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7573.774997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7575.198773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7580.960701] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7585.787647] LustreError: 239607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.816556] LustreError: 239607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7589.744142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.924594] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7589.975635] Lustre: 239597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7589.990499] Lustre: Permanently deactivating lustre-MDT0001 [ 7590.004683] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7590.399285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7594.911781] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7596.069925] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7603.133238] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7609.295374] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7615.492134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7615.782863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7620.911207] LustreError: 241449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7620.934937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7621.721044] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7626.213116] LustreError: 241449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7628.906422] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7631.333088] LustreError: 241452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.648726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7634.809263] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7635.840629] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7635.846750] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7635.865733] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7640.427798] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7647.835684] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7650.920823] Lustre: Permanently reactivating lustre-MDT0001 [ 7650.936982] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7650.942415] Lustre: Skipped 2 previous similar messages [ 7654.081146] Lustre: lustre-MDT0001: Client f80ff31d-773e-4655-9c9f-ad807da374c6 (at 192.168.201.49@tcp) reconnecting [ 7658.056840] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7658.065130] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7658.072214] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7658.079521] LustreError: 243318:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7658.080744] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7658.088293] LustreError: 243318:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7658.088523] LustreError: 243318:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.149@tcp: cfg command failed: rc = -108 [ 7658.092267] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7658.092267] [ 7658.108732] LustreError: 239600:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7659.567515] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7661.365706] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7666.392165] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7666.598223] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7671.286312] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7671.498381] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7674.303537] Lustre: Permanently deactivating lustre-MDT0001 [ 7674.328989] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7674.336983] Lustre: Skipped 2 previous similar messages [ 7683.192635] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7692.273654] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.285987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7692.304555] Lustre: Skipped 2 previous similar messages [ 7692.313817] Lustre: Skipped 3 previous similar messages [ 7697.385329] LustreError: 239601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.404406] LustreError: 239601:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7697.503926] Lustre: server umount lustre-MDT0000 complete [ 7701.140062] LustreError: 240346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681116 with bad export cookie 9551568853806336734 [ 7701.157120] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7701.158760] LustreError: 240346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7701.649841] Lustre: server umount lustre-MDT0001 complete [ 7707.925489] Lustre: server umount lustre-OST0000 complete [ 7722.481721] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:38:56 (1789681136) [ 7734.265763] Lustre: server umount lustre-OST0001 complete [ 7741.605898] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 7753.300103] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7787.924897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7788.135181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7788.140668] Lustre: Skipped 2 previous similar messages [ 7788.158134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7788.241527] Lustre: lustre-MDT0000: new disk, initializing [ 7788.307899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.329237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7792.696932] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7804.194273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.302915] Lustre: 247887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7804.358342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7804.361681] Lustre: Skipped 1 previous similar message [ 7804.464315] Lustre: lustre-MDT0001: new disk, initializing [ 7804.580060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7804.612601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7804.627262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7808.623778] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7817.137990] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7822.174940] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7828.980266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.229030] Lustre: lustre-OST0000: new disk, initializing [ 7829.232597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7829.237453] Lustre: 249855:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7829.308607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7835.577513] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7837.265288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7837.276226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7837.465648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7845.812474] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7848.192865] LustreError: 249864:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7855.810633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7856.022937] Lustre: lustre-OST0001: new disk, initializing [ 7856.028805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7856.039517] Lustre: 250969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7858.263087] LustreError: 249864:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7858.283113] LustreError: 249865:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7861.974773] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7865.952175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7865.966838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7866.091845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7868.311336] LustreError: 249865:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7870.693179] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7872.201783] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7876.586077] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7876.589777] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7876.592195] Lustre: Skipped 1 previous similar message [ 7876.593125] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7876.601478] Lustre: Skipped 1 previous similar message [ 7881.698574] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7881.703166] Lustre: Skipped 2 previous similar messages [ 7886.822360] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7886.828501] Lustre: Skipped 2 previous similar messages [ 7887.840115] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7887.982321] Lustre: server umount lustre-OST0001 complete [ 7891.942552] LustreError: 249838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7891.979930] LustreError: 249838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7904.230034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7904.237334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.238034] LustreError: Skipped 2 previous similar messages [ 7904.241400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7904.254381] Lustre: Skipped 2 previous similar messages [ 7904.269934] Lustre: Skipped 2 previous similar messages [ 7909.875127] Lustre: server umount lustre-OST0000 complete [ 7917.536466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.541131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7917.560887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7917.560897] Lustre: Skipped 2 previous similar messages [ 7919.523262] Lustre: server umount lustre-MDT0000 complete [ 7922.668271] LustreError: 247899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7922.691978] LustreError: 247899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7922.887087] LustreError: 249854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681337 with bad export cookie 9551568853806339772 [ 7922.894665] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7922.898455] LustreError: 249854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7929.435594] Lustre: server umount lustre-MDT0001 complete [ 7940.798581] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7943.397836] Key type lgssc unregistered [ 7943.665194] LNet: 253009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7943.671787] LNetError: 253009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7943.682522] LNet: Removed LNI 192.168.201.149@tcp [ 7944.430152] Key type .llcrypt unregistered [ 7944.432086] Key type ._llcrypt unregistered [ 7981.972781] Key type ._llcrypt registered [ 7981.974934] Key type .llcrypt registered [ 7982.059232] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7983.454782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7983.556432] alg: No test for adler32 (adler32-zlib) [ 7984.806369] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 7984.995416] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7986.663251] Key type lgssc registered [ 7987.733287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7995.653809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7995.676979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7996.972120] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7996.988702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7997.102873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8001.096149] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8009.221552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.287393] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8009.311671] Lustre: 255212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8009.540418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8014.209215] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8022.296566] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8027.567516] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8033.843741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8034.029774] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8034.232118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8039.971139] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8043.694272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8046.680944] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8100.328838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.334101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8100.341625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.353248] Lustre: Skipped 1 previous similar message [ 8101.647070] Lustre: server umount lustre-OST0000 complete [ 8105.628125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8116.749112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8117.057518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.103186] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8120.117266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8120.117609] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8120.134017] Lustre: Skipped 1 previous similar message [ 8123.150301] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8130.298555] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8171.492821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8171.498415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.509253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8171.514915] Lustre: Skipped 2 previous similar messages [ 8171.527953] Lustre: Skipped 1 previous similar message [ 8174.868346] Lustre: server umount lustre-OST0000 complete [ 8178.662670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8178.668454] LustreError: Skipped 1 previous similar message [ 8178.671564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.693700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8181.728318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8181.734074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.005916] Lustre: server umount lustre-MDT0000 complete [ 8186.987605] LustreError: 255206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681602 with bad export cookie 3349377860739222184 [ 8186.993444] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8186.997397] LustreError: 255206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8187.224951] Lustre: server umount lustre-MDT0001 complete [ 8196.422079] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8199.399729] Key type lgssc unregistered [ 8199.666909] LNet: 260632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8199.672829] LNetError: 260632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8199.684647] LNet: Removed LNI 192.168.201.149@tcp [ 8200.496817] Key type .llcrypt unregistered [ 8200.499437] Key type ._llcrypt unregistered [ 8221.642158] Key type ._llcrypt registered [ 8221.644101] Key type .llcrypt registered [ 8221.709268] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8222.745388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8222.778393] alg: No test for adler32 (adler32-zlib) [ 8223.779840] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 8224.018645] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8225.672257] Key type lgssc registered [ 8226.651136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8232.777392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8232.791617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8234.117793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8237.716585] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8244.372671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8244.752100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8248.888185] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8255.569326] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8260.579924] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8265.917574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8266.201803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8270.922675] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8277.404812] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8280.569872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8306.156038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8306.156247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8306.164972] LustreError: Skipped 1 previous similar message [ 8306.167385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8306.184614] Lustre: Skipped 1 previous similar message [ 8307.488671] Lustre: server umount lustre-OST0000 complete [ 8311.266105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8311.267712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8311.281577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8311.281596] Lustre: Skipped 1 previous similar message [ 8311.282783] Lustre: Skipped 1 previous similar message [ 8316.398417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8316.410316] Lustre: Skipped 2 previous similar messages [ 8321.505406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8321.520905] Lustre: Skipped 1 previous similar message [ 8325.087262] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8325.226561] Lustre: server umount lustre-MDT0000 complete [ 8326.625402] LustreError: 262050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8326.640828] LustreError: 262050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8328.317208] LustreError: 262029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681743 with bad export cookie 10418604925905953668 [ 8328.332063] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8328.332926] LustreError: 262029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8328.591704] Lustre: server umount lustre-MDT0001 complete [ 8338.364854] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8341.004650] Key type lgssc unregistered [ 8341.303672] LNet: 266352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8341.310871] LNetError: 266352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8342.383657] LNet: Removed LNI 192.168.201.149@tcp [ 8343.197709] Key type .llcrypt unregistered [ 8343.199100] Key type ._llcrypt unregistered [ 8357.844717] Key type ._llcrypt registered [ 8357.849637] Key type .llcrypt registered [ 8357.992292] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8359.315213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8359.366180] alg: No test for adler32 (adler32-zlib) [ 8360.356473] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 8360.588033] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8362.327301] Key type lgssc registered [ 8363.409904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8369.651731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8369.663031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.082840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8374.473646] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8381.416136] LustreError: 267550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8381.437161] LustreError: 267550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8382.586472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.963092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8387.447885] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8393.575603] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8398.147675] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8403.866694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.099172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.330276] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8415.655971] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8416.365465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8432.098867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.126315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8432.134796] Lustre: Skipped 1 previous similar message [ 8435.007950] Lustre: server umount lustre-OST0000 complete [ 8439.264910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8439.280282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8439.310167] Lustre: Skipped 1 previous similar message [ 8439.316704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8442.337277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8442.338473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8444.525259] Lustre: server umount lustre-MDT0000 complete [ 8447.457620] LustreError: 268277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8447.478199] LustreError: 268277:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8447.563040] LustreError: 268284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681862 with bad export cookie 5354153641066070947 [ 8447.569015] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8447.837507] Lustre: server umount lustre-MDT0001 complete [ 8455.948396] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8458.123745] Key type lgssc unregistered [ 8458.361744] LNet: 271224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8458.374235] LNetError: 271224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8458.388562] LNet: Removed LNI 192.168.201.149@tcp [ 8459.281157] Key type .llcrypt unregistered [ 8459.284571] Key type ._llcrypt unregistered [ 8479.162815] Key type ._llcrypt registered [ 8479.165340] Key type .llcrypt registered [ 8479.245831] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8479.933574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8480.035278] alg: No test for adler32 (adler32-zlib) [ 8481.020230] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 8481.190777] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8482.815300] Key type lgssc registered [ 8483.665690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8489.207176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8489.224387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.589724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8493.971987] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8500.712090] LustreError: 272615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.727835] LustreError: 272615:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8501.007754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8501.226289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8504.659369] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8510.902081] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8515.454391] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8521.001741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8521.239041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8526.742753] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8533.455922] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8535.608657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8561.129562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8561.131510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.136068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8561.154366] Lustre: Skipped 1 previous similar message [ 8563.400020] Lustre: server umount lustre-OST0000 complete [ 8567.776544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8567.783963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8567.803928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8567.807525] Lustre: Skipped 1 previous similar message [ 8571.365568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.368196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8571.381743] Lustre: Skipped 1 previous similar message [ 8572.561605] Lustre: server umount lustre-MDT0000 complete [ 8575.910935] LustreError: 272597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789681990 with bad export cookie 7714948144802831833 [ 8575.928461] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8575.928953] LustreError: 272597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8576.219486] Lustre: server umount lustre-MDT0001 complete [ 8584.919673] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8586.951518] Key type lgssc unregistered [ 8587.169886] LNet: 276918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8587.174502] LNetError: 276918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8587.191712] LNet: Removed LNI 192.168.201.149@tcp [ 8587.867966] Key type .llcrypt unregistered [ 8587.870112] Key type ._llcrypt unregistered [ 8600.339464] Key type ._llcrypt registered [ 8600.343445] Key type .llcrypt registered [ 8600.410137] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8601.018446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8601.053463] alg: No test for adler32 (adler32-zlib) [ 8602.050264] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 8602.216885] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8603.839578] Key type lgssc registered [ 8604.552962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8609.482758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8609.505966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8610.931794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8613.784769] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8619.467933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8619.771172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8622.719535] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8628.017162] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8631.845423] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8636.725326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8637.045409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8641.217755] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8646.309786] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8655.471635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8666.090901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.112699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8666.117258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8666.125249] Lustre: Skipped 1 previous similar message [ 8670.798568] Lustre: server umount lustre-OST0000 complete [ 8676.320301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8676.323372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.324776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8676.329333] Lustre: Skipped 2 previous similar messages [ 8679.845641] Lustre: server umount lustre-MDT0000 complete [ 8681.440254] LustreError: 278139:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8681.449798] LustreError: 278139:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8682.134715] LustreError: 278095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682097 with bad export cookie 13900092076152868130 [ 8682.143559] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8682.372687] Lustre: server umount lustre-MDT0001 complete [ 8689.200699] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8691.191816] Key type lgssc unregistered [ 8691.379407] LNet: 281842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8691.384357] LNetError: 281842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8691.404638] LNet: Removed LNI 192.168.201.149@tcp [ 8692.037504] Key type .llcrypt unregistered [ 8692.040892] Key type ._llcrypt unregistered [ 8698.856187] Key type ._llcrypt registered [ 8698.857969] Key type .llcrypt registered [ 8698.934128] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8709.603581] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 8719.060147] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8719.576821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8719.594116] alg: No test for adler32 (adler32-zlib) [ 8720.541681] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 8720.695530] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8722.311249] Key type lgssc registered [ 8723.055284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8745.707835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8745.740667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8746.873516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8746.893565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8746.949868] Lustre: lustre-MDT0000: new disk, initializing [ 8746.994276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.002442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8749.863427] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8756.921378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8757.014514] Lustre: 285366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8757.050755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8757.053447] Lustre: Skipped 1 previous similar message [ 8757.110950] Lustre: lustre-MDT0001: new disk, initializing [ 8757.180016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8757.206821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8757.216958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8759.937800] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8766.559671] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8770.300659] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8775.260864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8775.495858] Lustre: lustre-OST0000: new disk, initializing [ 8775.498766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8775.503516] Lustre: 287332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8775.595581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8779.802643] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8783.934664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8783.941699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8783.994515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8787.246753] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8790.091464] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8790.254165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8793.531814] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8793.670180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8799.210571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8799.213893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8799.228943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8799.235751] Lustre: Skipped 1 previous similar message [ 8801.219680] Lustre: server umount lustre-OST0000 complete [ 8804.324499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.328863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8804.332092] Lustre: Skipped 2 previous similar messages [ 8804.346707] Lustre: Skipped 1 previous similar message [ 8809.444185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8809.447709] Lustre: Skipped 1 previous similar message [ 8814.561460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8814.566657] Lustre: Skipped 1 previous similar message [ 8818.143318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8818.266589] Lustre: server umount lustre-MDT0000 complete [ 8820.472898] LustreError: 285358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682235 with bad export cookie 14957800477311947746 [ 8820.478891] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8820.484852] LustreError: 285358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8820.690781] Lustre: server umount lustre-MDT0001 complete [ 8826.276952] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:57:21 (1789682241) [ 8831.148433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8831.493700] LustreError: 289745:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.510610] LustreError: 289745:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8831.556762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8834.477957] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8836.579370] LustreError: 289746:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8840.181930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8840.539798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8843.506082] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8849.279579] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8853.287835] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8858.737450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8859.000321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8863.537761] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8869.060424] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8885.731346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8885.739608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8885.749290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.762099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8891.163297] Lustre: server umount lustre-OST0000 complete [ 8893.746251] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8893.754609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8893.769094] Lustre: Skipped 1 previous similar message [ 8893.782190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8893.788157] Lustre: Skipped 2 previous similar messages [ 8908.255131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8908.387401] Lustre: server umount lustre-MDT0000 complete [ 8910.998472] LustreError: 289725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682326 with bad export cookie 14957800477311948628 [ 8911.005167] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8911.006262] LustreError: 290481:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 8911.007171] LustreError: 289725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8911.259400] Lustre: server umount lustre-MDT0001 complete [ 8918.680661] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8920.352585] Key type lgssc unregistered [ 8920.599905] LNet: 293299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8920.603389] LNetError: 293299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8920.623246] LNet: Removed LNI 192.168.201.149@tcp [ 8921.229229] Key type .llcrypt unregistered [ 8921.231064] Key type ._llcrypt unregistered [ 8938.886359] Key type ._llcrypt registered [ 8938.888759] Key type .llcrypt registered [ 8938.957311] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 8949.010237] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8949.581159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8949.680954] alg: No test for adler32 (adler32-zlib) [ 8950.686135] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 8950.888432] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8952.548534] Key type lgssc registered [ 8953.351586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8980.444755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8980.463349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8981.573624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8981.584615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8981.634914] Lustre: lustre-MDT0000: new disk, initializing [ 8981.664816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8981.673466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8984.138457] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8990.723812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.791240] Lustre: 297098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8990.820610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8990.822616] Lustre: Skipped 1 previous similar message [ 8990.867699] Lustre: lustre-MDT0001: new disk, initializing [ 8990.915712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8990.945465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8990.951858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8993.404537] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8999.317477] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9002.639871] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9007.415991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9007.598645] Lustre: lustre-OST2710: new disk, initializing [ 9007.601638] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9007.606214] Lustre: 299062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9007.674445] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9011.519537] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9016.336297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9016.344431] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9016.414778] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9018.075589] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9022.371713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9022.461854] Lustre: lustre-OST03e8: new disk, initializing [ 9022.464178] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9022.467434] Lustre: 300128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9022.524162] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9024.936322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9024.941240] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9024.986532] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9026.114922] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9030.405716] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9035.150685] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9035.335991] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9037.963747] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9038.091306] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9040.618296] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9040.753779] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9043.767664] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9043.922444] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9047.520535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9047.527599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9047.536299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.596820] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.597586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.604660] Lustre: Skipped 2 previous similar messages [ 9050.607686] Lustre: Skipped 2 previous similar messages [ 9052.720470] Lustre: server umount lustre-MDT0000 complete [ 9055.149672] LustreError: 297089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682470 with bad export cookie 15016267076181765119 [ 9055.153368] LustreError: 297089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9055.154148] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9055.422721] Lustre: server umount lustre-MDT0001 complete [ 9068.351024] Lustre: server umount lustre-OST2710 complete [ 9080.914539] Lustre: server umount lustre-OST03e8 complete [ 9090.913798] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 9096.502020] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9118.322889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9118.439861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9118.456704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9118.510272] Lustre: lustre-MDT0000: new disk, initializing [ 9118.544197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9118.563250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9121.268001] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9127.162803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9127.218254] Lustre: 305382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9127.248344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9127.250850] Lustre: Skipped 1 previous similar message [ 9127.301661] Lustre: lustre-MDT0001: new disk, initializing [ 9127.368943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9127.391678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9127.395833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9129.514764] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9134.927437] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9138.019290] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9142.225699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9142.383773] Lustre: 307350:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9143.178944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9143.246614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9145.761956] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9151.135955] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9153.671159] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9153.779433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9156.233382] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9156.355286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9158.627676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9158.629161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9158.632411] LustreError: Skipped 1 previous similar message [ 9158.633545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9158.633552] Lustre: Skipped 1 previous similar message [ 9158.655180] Lustre: Skipped 1 previous similar message [ 9163.500774] Lustre: server umount lustre-OST0000 complete [ 9168.354932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9168.359488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.368112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9168.371026] Lustre: Skipped 1 previous similar message [ 9171.565453] Lustre: server umount lustre-MDT0000 complete [ 9173.213262] LustreError: 305373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682588 with bad export cookie 15016267076181766806 [ 9173.215233] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.219241] LustreError: 305373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9173.358331] Lustre: server umount lustre-MDT0001 complete [ 9177.183267] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:03:12 (1789682592) [ 9177.808419] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9178.484898] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:03:13 (1789682593) [ 9179.690636] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9192.048967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.193239] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9192.205983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9192.295430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.300530] Lustre: Skipped 1 previous similar message [ 9194.408453] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9199.390944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9199.443858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9199.469608] Lustre: 311199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9202.121416] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9220.835879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9220.928820] Lustre: Denying initial registration attempt from nid 192.168.201.149@tcp, specified as failover [ 9220.936084] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9220.940561] LustreError: 312655:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9220.944355] LustreError: 312655:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9220.947075] LustreError: 312655:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9220.973865] Lustre: server umount lustre-OST0000 complete [ 9220.975594] LustreError: 312655:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9225.185425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9225.193205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.202330] Lustre: Skipped 1 previous similar message [ 9225.205349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9225.208398] Lustre: Skipped 1 previous similar message [ 9230.189298] Lustre: server umount lustre-MDT0000 complete [ 9232.245450] LustreError: 311194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682647 with bad export cookie 15016267076181767681 [ 9232.251553] LustreError: 311194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9232.256712] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9232.432373] Lustre: server umount lustre-MDT0001 complete [ 9238.053280] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 9239.669192] Key type lgssc unregistered [ 9239.848678] LNet: 313775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9239.852666] LNetError: 313775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9239.860624] LNet: Removed LNI 192.168.201.149@tcp [ 9240.314886] Key type .llcrypt unregistered [ 9240.316506] Key type ._llcrypt unregistered [ 9246.521517] Key type ._llcrypt registered [ 9246.523994] Key type .llcrypt registered [ 9246.589044] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9247.188294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9247.200533] alg: No test for adler32 (adler32-zlib) [ 9248.126484] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 9248.279357] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 9249.896090] Key type lgssc registered [ 9250.444391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9279.265634] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9282.970978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9282.981016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.085817] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9284.098788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9284.188811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.973100] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9289.793829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.837520] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9289.851738] Lustre: 316510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9289.981789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9292.172940] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9295.936469] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9299.249626] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9316.390387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9316.529716] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9316.637236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9319.479481] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9322.838925] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9327.074132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9327.074533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9327.077769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9327.081215] LustreError: Skipped 1 previous similar message [ 9327.085552] Lustre: Skipped 1 previous similar message [ 9330.046871] Lustre: server umount lustre-OST0000 complete [ 9332.195835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.200958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9332.202143] Lustre: Skipped 1 previous similar message [ 9332.209974] Lustre: Skipped 2 previous similar messages [ 9337.315230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.093698] Lustre: server umount lustre-MDT0000 complete [ 9340.000353] LustreError: 318373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682755 with bad export cookie 15577053765304409157 [ 9340.008623] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9340.175957] Lustre: server umount lustre-MDT0001 complete [ 9345.155424] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 9346.564230] Key type lgssc unregistered [ 9346.715402] LNet: 320079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9346.718761] LNetError: 320079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9346.729864] LNet: Removed LNI 192.168.201.149@tcp [ 9347.145620] Key type .llcrypt unregistered [ 9347.146907] Key type ._llcrypt unregistered [ 9359.390381] Key type ._llcrypt registered [ 9359.391568] Key type .llcrypt registered [ 9359.436255] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9359.903040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9359.970600] alg: No test for adler32 (adler32-zlib) [ 9360.860363] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 9360.974271] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 9362.559221] Key type lgssc registered [ 9363.016811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9366.459412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9366.469992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9367.666789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9369.465692] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9373.211038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9373.370629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9375.350956] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9378.467570] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9380.754059] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9385.572622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9385.729161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9388.235586] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9391.451981] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9401.234816] LustreError: 324043:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0xf14dc48e (00000000371e3955) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9403.272810] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 9403.360992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9403.441389] Lustre: server umount lustre-MDT0000 complete [ 9405.318996] LustreError: 321449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682820 with bad export cookie 7418907583724943682 [ 9405.322471] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9405.328061] LustreError: 321449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9408.381164] LustreError: 321492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9408.398888] LustreError: 321492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9408.399980] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 9408.407552] Lustre: Skipped 3 previous similar messages [ 9408.481778] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9408.486420] Lustre: Skipped 2 previous similar messages [ 9410.528059] LustreError: 321463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9410.528286] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9410.534527] LustreError: 321463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9410.537331] Lustre: Skipped 1 previous similar message [ 9411.557601] Lustre: server umount lustre-MDT0001 complete [ 9415.523641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9420.000439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9429.984042] LustreError: 320778:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005cfb9ec2 x1876618406551552/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9430.119441] LustreError: 323296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9430.152120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9430.177468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9431.952648] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9435.299103] LustreError: 323297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.308694] LustreError: 323297:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9436.029284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.250302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9438.287751] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9440.359034] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9441.999303] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9445.786578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 9448.147439] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9449.901292] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 9456.099872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9456.102260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9456.111834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9456.115395] Lustre: Skipped 1 previous similar message [ 9457.701352] Lustre: server umount lustre-OST0000 complete [ 9461.216653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.217602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9461.224151] Lustre: Skipped 2 previous similar messages [ 9461.229438] Lustre: Skipped 1 previous similar message [ 9465.731933] Lustre: server umount lustre-MDT0000 complete [ 9466.338856] LustreError: 325057:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9466.345799] LustreError: 325057:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9467.819839] LustreError: 323308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682882 with bad export cookie 7418907583724950836 [ 9467.821710] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9467.828665] LustreError: 323308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9467.992253] Lustre: server umount lustre-MDT0001 complete [ 9472.632242] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 9473.873843] Key type lgssc unregistered [ 9474.045456] LNet: 327724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9474.049293] LNetError: 327724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9474.059392] LNet: Removed LNI 192.168.201.149@tcp [ 9474.456959] Key type .llcrypt unregistered [ 9474.458440] Key type ._llcrypt unregistered [ 9486.033220] Key type ._llcrypt registered [ 9486.034366] Key type .llcrypt registered [ 9486.083792] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9486.401429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9486.413939] alg: No test for adler32 (adler32-zlib) [ 9487.309674] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 9487.423196] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 9489.031203] Key type lgssc registered [ 9489.558207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9493.039713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9493.048173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9494.276932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.304697] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9501.005720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.247158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9503.481230] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9506.990084] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9509.339158] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9511.392432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9511.395518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9511.407906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9512.418606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9512.420639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9517.538021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9517.544875] Lustre: Skipped 2 previous similar messages [ 9522.657480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9522.663307] Lustre: Skipped 1 previous similar message [ 9525.215229] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9525.289151] Lustre: server umount lustre-MDT0000 complete [ 9526.936335] LustreError: 330735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789682942 with bad export cookie 622085545892222996 [ 9526.939822] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9526.941825] LustreError: 330735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9527.075394] Lustre: server umount lustre-MDT0001 complete [ 9533.544743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9533.668633] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9533.682057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9533.778816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9535.513871] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9539.454690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9539.509144] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9539.525370] Lustre: 331703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9539.661380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9541.559941] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9545.153459] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9548.189409] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9554.485062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9554.644710] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9554.650898] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9554.656446] LustreError: 333587:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9554.661259] LustreError: 333587:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9554.665251] LustreError: 333587:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9554.708199] Lustre: server umount lustre-OST0000 complete [ 9554.710708] LustreError: 333587:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9561.514568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9561.662171] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9561.801247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9564.836817] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9568.525225] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9569.834114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9572.235868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9572.274764] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9572.312673] Lustre: lustre-OST0001: new disk, initializing [ 9572.314959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9572.317628] Lustre: 335127:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9572.364297] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9575.138393] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9576.972762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9576.977623] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9577.023200] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9578.423592] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9582.049085] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9582.051726] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9582.053331] LustreError: Skipped 1 previous similar message [ 9582.060493] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9582.064492] Lustre: Skipped 1 previous similar message [ 9585.829049] Lustre: server umount lustre-OST0001 complete [ 9587.177568] LustreError: 334055:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9587.183446] LustreError: 334055:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9590.240757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9590.240822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.245568] LustreError: Skipped 1 previous similar message [ 9590.250769] Lustre: Skipped 1 previous similar message [ 9590.253288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9590.256299] Lustre: Skipped 1 previous similar message [ 9592.292296] LustreError: 334834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.298108] LustreError: 334834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9593.835683] Lustre: server umount lustre-OST0000 complete [ 9595.872225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9595.875466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.698463] Lustre: server umount lustre-MDT0000 complete [ 9602.528173] LustreError: 333020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9602.535579] LustreError: 333020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9603.183053] LustreError: 331695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683018 with bad export cookie 622085545892223535 [ 9603.185543] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9603.192815] LustreError: 331695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9603.340690] Lustre: server umount lustre-MDT0001 complete [ 9607.641212] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 9608.846813] Key type lgssc unregistered [ 9608.986283] LNet: 337083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9608.990067] LNetError: 337083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9608.998413] LNet: Removed LNI 192.168.201.149@tcp [ 9609.346782] Key type .llcrypt unregistered [ 9609.348489] Key type ._llcrypt unregistered [ 9651.235345] Key type ._llcrypt registered [ 9651.237274] Key type .llcrypt registered [ 9651.348174] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 9667.559515] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9669.645680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9669.662733] alg: No test for adler32 (adler32-zlib) [ 9670.843236] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 9671.225889] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 9672.928122] Key type lgssc registered [ 9674.316294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9708.537794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9708.609134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9709.986172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9710.050227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9710.169956] Lustre: lustre-MDT0000: new disk, initializing [ 9710.280278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9710.294253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9715.564239] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9727.310301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9727.428456] Lustre: 342008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9727.476415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9727.480092] Lustre: Skipped 1 previous similar message [ 9727.545312] Lustre: lustre-MDT0001: new disk, initializing [ 9727.609282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9727.642847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9727.652280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9731.537340] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9740.055072] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9745.177703] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9751.365517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9751.563631] Lustre: lustre-OST0000: new disk, initializing [ 9751.566594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9751.574125] Lustre: 343974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9751.652855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9756.677437] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9760.345588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9760.353042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9760.444760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9765.794882] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9769.922152] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9770.136686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9774.110728] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9774.304465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9776.112518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9776.114430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9776.124655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9776.159209] Lustre: Skipped 1 previous similar message [ 9781.222429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9781.228943] Lustre: Skipped 2 previous similar messages [ 9782.176563] Lustre: server umount lustre-OST0000 complete [ 9786.341589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.342354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9786.348143] Lustre: Skipped 1 previous similar message [ 9791.456586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9791.465889] Lustre: Skipped 2 previous similar messages [ 9796.576553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9796.583599] Lustre: Skipped 1 previous similar message [ 9799.135144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9799.271092] Lustre: server umount lustre-MDT0000 complete [ 9801.977668] LustreError: 341999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683217 with bad export cookie 8073031203531533210 [ 9801.983496] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9801.989385] LustreError: 341999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9802.205951] Lustre: server umount lustre-MDT0001 complete [ 9809.019994] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:13:43 (1789683223) [ 9814.338546] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:13:49 (1789683229) [ 9821.039467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.317534] LustreError: 346732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9821.329363] LustreError: 346732:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9821.384646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9825.861925] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9826.800974] LustreError: 346733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.919471] LustreError: 346732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.034607] LustreError: 346733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.340409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9837.808461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9845.121107] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9860.045725] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9870.939703] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9883.967611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9884.777789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9893.518608] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9902.132906] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9915.513593] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9918.387360] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9920.360765] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9922.678360] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9925.228540] Lustre: Failing over lustre-MDT0000 [ 9925.569422] Lustre: server umount lustre-MDT0000 complete [ 9926.051283] LustreError: 346727:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.167173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.182716] Lustre: Skipped 1 previous similar message [ 9936.270203] LustreError: 346729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9936.302119] LustreError: 346729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 9944.545399] Lustre: 340234:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789683343/real 1789683343] req@000000003c0e5beb x1876618731661312/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789683359 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9944.570967] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9945.756379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9954.785059] LustreError: 340230:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006f60d475 x1876618731666688/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9954.823133] LustreError: 347652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9954.846603] LustreError: 347652:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 9955.135366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9955.185732] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9956.741969] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9959.600587] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9960.472254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9960.512984] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9960.601330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9971.392316] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9973.514829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9977.717329] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9980.425151] Lustre: Failing over lustre-MDT0000 [ 9980.898865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9980.901935] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9980.908447] LustreError: Skipped 1 previous similar message [ 9980.940613] Lustre: Skipped 3 previous similar messages [ 9980.954757] Lustre: server umount lustre-MDT0000 complete [ 9987.452803] LustreError: 346729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9987.464893] LustreError: 346729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10001.842088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.935392] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10002.177450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10002.256621] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10003.834666] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10007.539349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10007.547085] Lustre: Skipped 2 previous similar messages [10007.580707] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10007.632093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10008.108388] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10020.991934] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10023.009412] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10033.120883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10033.121947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10033.126412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10033.143542] Lustre: Skipped 1 previous similar message [10036.885802] Lustre: server umount lustre-MDT0000 complete [10040.919972] LustreError: 346712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683456 with bad export cookie 8073031203531535919 [10040.932862] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10040.933807] LustreError: 346712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10041.415283] Lustre: server umount lustre-MDT0001 complete [10056.757822] Lustre: server umount lustre-OST0000 complete [10075.319868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10076.257631] LustreError: 353098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10076.284327] LustreError: 353098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [10076.424781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10081.952932] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10092.506278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10098.367264] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10107.797934] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10115.214129] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10122.485702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10130.625392] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10139.449407] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10146.550957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10157.035173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10157.040060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.057624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10157.064979] Lustre: Skipped 2 previous similar messages [10157.076868] Lustre: Skipped 2 previous similar messages [10162.638801] Lustre: server umount lustre-OST0000 complete [10169.824926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10173.765431] Lustre: server umount lustre-MDT0000 complete [10177.716825] LustreError: 353080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683592 with bad export cookie 8073031203531536661 [10177.726450] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10178.159482] Lustre: server umount lustre-MDT0001 complete [10192.303546] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [10195.539812] Key type lgssc unregistered [10195.981828] LNet: 356653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10195.996557] LNetError: 356653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10196.030457] LNet: Removed LNI 192.168.201.149@tcp [10196.870188] Key type .llcrypt unregistered [10196.876960] Key type ._llcrypt unregistered [10228.675811] Key type ._llcrypt registered [10228.678649] Key type .llcrypt registered [10228.743419] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10229.908730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10230.034586] alg: No test for adler32 (adler32-zlib) [10231.300826] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [10231.569565] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [10233.289869] Key type lgssc registered [10234.610081] Lustre: Echo OBD driver; http://www.lustre.org/ [10243.561901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10243.594205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.314404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10250.675420] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10255.841874] LustreError: 358169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.878413] LustreError: 358169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10260.970775] LustreError: 358170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10262.118848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10262.662303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10269.184675] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10280.946959] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10288.021253] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10295.404349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10295.787058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10302.874139] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10312.975110] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10315.254233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10334.179462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10334.208177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10334.247991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10335.717638] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10335.719740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10335.785883] Lustre: Skipped 2 previous similar messages [10340.342343] Lustre: server umount lustre-MDT0000 complete [10340.836119] LustreError: 358170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.865355] LustreError: 358170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10345.138637] LustreError: 358150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683760 with bad export cookie 16613042785885493791 [10345.145920] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10345.162378] LustreError: 358150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10345.780197] Lustre: server umount lustre-MDT0001 complete [10361.585240] Lustre: server umount lustre-OST0000 complete [10378.514621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10379.155849] LustreError: 361907:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.255553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10384.082497] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10384.356974] LustreError: 361908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10394.077723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.426743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10399.037806] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10409.661733] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10417.545433] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10428.495036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10429.296046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10438.732805] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10449.012341] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10449.775830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10470.388276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10470.394764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10470.417612] Lustre: Skipped 1 previous similar message [10470.418161] Lustre: Skipped 3 previous similar messages [10475.060527] Lustre: server umount lustre-MDT0000 complete [10475.494093] LustreError: 361903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10475.510157] LustreError: 361903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10479.693116] LustreError: 361888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683894 with bad export cookie 16613042785885494904 [10479.699392] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10479.711425] LustreError: 361888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10480.253898] Lustre: server umount lustre-MDT0001 complete [10495.720163] Lustre: server umount lustre-OST0000 complete [10515.112793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.503314] LustreError: 365821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10515.593277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10520.445766] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10531.193364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10538.028866] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10547.919039] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10557.086610] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10566.544164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10567.075648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10567.084630] Lustre: Skipped 1 previous similar message [10576.242666] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10585.540383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10587.202727] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10595.827966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10595.845745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10595.850096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10595.850766] Lustre: Skipped 1 previous similar message [10599.886611] Lustre: server umount lustre-OST0000 complete [10606.065982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10606.093251] Lustre: Skipped 2 previous similar messages [10606.095652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10606.105283] Lustre: Skipped 2 previous similar messages [10610.977249] Lustre: server umount lustre-MDT0000 complete [10611.186506] LustreError: 365822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.218548] LustreError: 365822:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10616.198279] LustreError: 365802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684031 with bad export cookie 16613042785885495926 [10616.200866] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10616.215229] LustreError: 365802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10616.951936] Lustre: server umount lustre-MDT0001 complete [10633.787215] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [10638.202173] Key type lgssc unregistered [10638.622889] LNet: 369356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10638.631259] LNetError: 369356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10638.657266] LNet: Removed LNI 192.168.201.149@tcp [10640.167552] Key type .llcrypt unregistered [10640.173561] Key type ._llcrypt unregistered [10676.524558] Key type ._llcrypt registered [10676.526086] Key type .llcrypt registered [10676.619248] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10678.378407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10678.744141] alg: No test for adler32 (adler32-zlib) [10679.751644] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [10680.027216] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [10681.695764] Key type lgssc registered [10683.127520] Lustre: Echo OBD driver; http://www.lustre.org/ [10693.166503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10693.245785] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10693.259896] LustreError: 370798:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10693.291315] LustreError: 370798:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10693.303710] LustreError: 370798:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10693.314726] LustreError: 370798:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10693.336145] LustreError: 370798:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10702.726555] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10702.732807] LustreError: 371195:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10702.746573] LustreError: 371195:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10702.760644] LustreError: 371195:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10702.766514] LustreError: 371195:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10702.774730] LustreError: 371195:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10725.482777] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [10728.584716] Key type lgssc unregistered [10728.846636] LNet: 372198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10728.857333] LNetError: 372198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10728.899374] LNet: Removed LNI 192.168.201.149@tcp [10729.739404] Key type .llcrypt unregistered [10729.741855] Key type ._llcrypt unregistered [10749.104820] Key type ._llcrypt registered [10749.113611] Key type .llcrypt registered [10749.238312] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [10767.589469] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10768.900736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10768.938927] alg: No test for adler32 (adler32-zlib) [10770.006435] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [10770.240335] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [10771.919222] Key type lgssc registered [10773.030049] Lustre: Echo OBD driver; http://www.lustre.org/ [10806.296959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10806.346234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.838560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10807.895712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10807.955149] Lustre: lustre-MDT0000: new disk, initializing [10808.051897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10808.070646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10813.952452] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10827.058854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10827.286882] Lustre: 375353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10827.375684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10827.380937] Lustre: Skipped 1 previous similar message [10827.490356] Lustre: lustre-MDT0001: new disk, initializing [10827.619630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10827.713440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10827.750423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10834.560646] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10848.206340] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10856.312394] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10863.094375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10863.386992] Lustre: lustre-OST0000: new disk, initializing [10863.390500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10863.397091] Lustre: 377324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10863.508735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10869.915874] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10872.386435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10872.413824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10872.487962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10882.971739] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10887.883588] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10888.142734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10893.506406] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10893.738596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10897.895975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.905662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10897.919521] Lustre: Skipped 1 previous similar message [10897.947524] Lustre: Skipped 1 previous similar message [10903.024592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10903.033426] Lustre: Skipped 1 previous similar message [10908.165450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10908.177094] Lustre: Skipped 1 previous similar message [10910.175329] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10910.453445] Lustre: server umount lustre-OST0000 complete [10918.368265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.370460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10918.390659] Lustre: Skipped 1 previous similar message [10923.492145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10923.496715] Lustre: Skipped 2 previous similar messages [10929.121482] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10929.376301] Lustre: server umount lustre-MDT0000 complete [10933.994979] LustreError: 377323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684349 with bad export cookie 7954685994449313181 [10934.004757] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10934.009361] LustreError: 377323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10934.792626] Lustre: server umount lustre-MDT0001 complete [10944.836211] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10947.413753] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:32:41 (1789684361) [10956.362471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.321468] LustreError: 379786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10957.355924] LustreError: 379786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10957.489494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10962.706825] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10962.912854] LustreError: 379787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10967.011261] LustreError: 379786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10972.134678] LustreError: 379787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10972.991330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10973.556486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10978.182188] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10986.556789] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10993.822845] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11002.034044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11002.342521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11008.661727] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11016.553521] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11024.249636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11024.472369] Lustre: lustre-OST0001: new disk, initializing [11024.480386] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11024.491765] Lustre: 382626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11024.632915] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11030.237125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11030.248738] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11030.383535] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11031.147558] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11041.686549] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11045.856252] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11045.861408] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11045.871738] LustreError: Skipped 1 previous similar message [11045.874491] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11045.881542] Lustre: Skipped 1 previous similar message [11045.904409] Lustre: Skipped 3 previous similar messages [11051.804273] Lustre: server umount lustre-OST0001 complete [11053.949302] LustreError: 381608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11066.344611] LustreError: 382586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11066.375374] LustreError: 382586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11069.923723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11069.937110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11069.943661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11069.947359] Lustre: Skipped 3 previous similar messages [11084.255366] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11084.623891] Lustre: server umount lustre-OST0000 complete [11091.430988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11091.434883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.440871] Lustre: Skipped 1 previous similar message [11094.815233] Lustre: server umount lustre-MDT0000 complete [11097.056193] LustreError: 379786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11097.065326] LustreError: 379786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [11099.293629] LustreError: 382624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684514 with bad export cookie 7954685994449314063 [11099.301798] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11099.314782] LustreError: 382624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11105.896465] Lustre: server umount lustre-MDT0001 complete [11119.325749] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [11122.486775] Key type lgssc unregistered [11122.783678] LNet: 384610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11122.795593] LNetError: 384610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11123.814418] LNet: Removed LNI 192.168.201.149@tcp [11124.906550] Key type .llcrypt unregistered [11124.909064] Key type ._llcrypt unregistered [11179.549325] Key type ._llcrypt registered [11179.551668] Key type .llcrypt registered [11179.681278] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [11180.805097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11180.902498] alg: No test for adler32 (adler32-zlib) [11181.922210] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [11182.189542] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [11183.913700] Key type lgssc registered [11184.919249] Lustre: Echo OBD driver; http://www.lustre.org/ [11193.378783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11193.403500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.641533] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11194.660142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11194.785844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11199.548626] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11207.711495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11207.789174] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11207.820760] Lustre: 387405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11208.051473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11212.266515] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11219.789714] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11225.596279] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11228.643828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11228.648472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11228.657420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11230.176229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11230.177338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11233.860363] Lustre: server umount lustre-MDT0000 complete [11236.618931] LustreError: 387396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684651 with bad export cookie 13063503017075540524 [11236.623311] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11236.629058] LustreError: 387396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11236.883346] Lustre: server umount lustre-MDT0001 complete [11245.763929] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:37:40 (1789684660) [11251.823066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.399313] LustreError: 389921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.417965] LustreError: 389921:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11252.479333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11256.686236] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11257.826925] LustreError: 389922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.946219] LustreError: 389921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.717501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11266.144210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11270.691372] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11279.638543] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11286.817888] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11293.740968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11293.982838] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11294.189658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11300.940679] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11308.672162] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11316.350146] Lustre: Permanently deactivating lustre-OST0000 [11316.376549] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11323.953680] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11325.580526] LustreError: 392908:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11325.589917] LustreError: 392908:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11325.601314] LustreError: 392908:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11328.134626] LustreError: 392956:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11328.139199] LustreError: 392956:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11336.038933] Lustre: server umount lustre-OST0000 complete [11338.510764] LustreError: 393206:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11338.515240] LustreError: 393206:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11341.282890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11341.300076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11341.308138] Lustre: Skipped 2 previous similar messages [11346.409844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11346.420104] Lustre: Skipped 1 previous similar message [11350.583650] Lustre: server umount lustre-MDT0000 complete [11351.522626] LustreError: 389922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11351.538535] LustreError: 389922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11354.306994] LustreError: 389901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684769 with bad export cookie 13063503017075541105 [11354.318198] LustreError: 389901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11354.324338] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11364.953912] Lustre: server umount lustre-MDT0001 complete [11373.232236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11377.891809] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11380.385839] LustreError: 394364:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11385.687331] Lustre: 394704:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.149@tcp [11387.541178] Lustre: server umount MGS complete [11396.444635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11396.803642] LustreError: 395429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11396.827667] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11396.833167] Lustre: Skipped 1 previous similar message [11396.891612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11400.950721] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11407.330967] LustreError: 395430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11407.354522] LustreError: 395430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11407.720779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11407.928269] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11411.509936] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11417.682445] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11423.006483] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11424.875159] Lustre: Permanently reactivating lustre-OST0000 [11424.888724] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11424.893794] Lustre: Skipped 2 previous similar messages [11455.455317] LustreError: 397300:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11463.339859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11463.706675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11463.713213] Lustre: Skipped 1 previous similar message [11468.778989] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11475.583799] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11481.058925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11481.070741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11481.074534] Lustre: Skipped 2 previous similar messages [11481.083675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11486.473162] Lustre: server umount lustre-OST0000 complete [11491.298291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11491.302802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11491.312460] Lustre: Skipped 1 previous similar message [11491.331748] Lustre: Skipped 4 previous similar messages [11496.442796] LustreError: 397403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11496.463171] LustreError: 397403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11496.562402] Lustre: server umount lustre-MDT0000 complete [11499.926055] LustreError: 395410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684915 with bad export cookie 13063503017075542155 [11499.931544] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11499.938394] LustreError: 395410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11500.202724] Lustre: server umount lustre-MDT0001 complete [11509.062701] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [11511.158501] Key type lgssc unregistered [11511.384137] LNet: 399933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11511.388460] LNetError: 399933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11511.401980] LNet: Removed LNI 192.168.201.149@tcp [11512.013723] Key type .llcrypt unregistered [11512.015691] Key type ._llcrypt unregistered [11525.773719] Key type ._llcrypt registered [11525.775637] Key type .llcrypt registered [11525.848394] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [11536.519955] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [11537.460918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11537.473864] alg: No test for adler32 (adler32-zlib) [11538.459251] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [11538.613574] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [11540.255178] Key type lgssc registered [11541.003838] Lustre: Echo OBD driver; http://www.lustre.org/ [11567.450719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11567.503014] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11568.692194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11568.713668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11568.765464] Lustre: lustre-MDT0000: new disk, initializing [11568.819975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11568.831892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11571.992913] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11579.906352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11580.043698] Lustre: 403063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11580.106920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11580.109274] Lustre: Skipped 1 previous similar message [11580.157254] Lustre: lustre-MDT0001: new disk, initializing [11580.248155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11580.298180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11580.309947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11584.846232] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11593.525942] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11598.390671] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11603.916251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11604.088149] Lustre: lustre-OST0000: new disk, initializing [11604.090605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11604.094360] Lustre: 405033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11604.136126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11608.434076] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11610.660991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11610.670916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11610.754261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11615.877927] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11619.548813] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11619.703464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11622.994914] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11623.194440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11626.465021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11626.471075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11626.473831] LustreError: Skipped 1 previous similar message [11626.478599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11626.490336] Lustre: Skipped 1 previous similar message [11631.586322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11631.608426] Lustre: Skipped 2 previous similar messages [11636.717337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11636.722347] Lustre: Skipped 1 previous similar message [11639.263510] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11639.403418] Lustre: server umount lustre-OST0000 complete [11646.944231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11646.947685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11646.948435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11646.965917] Lustre: Skipped 1 previous similar message [11649.452324] Lustre: server umount lustre-MDT0000 complete [11652.387129] LustreError: 403057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685067 with bad export cookie 5679230046900524859 [11652.389835] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11652.399016] LustreError: 403057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11652.733740] Lustre: server umount lustre-MDT0001 complete [11660.208351] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11661.923368] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:44:36 (1789685076) [11667.414251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.782112] LustreError: 407494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11667.803917] LustreError: 407494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11667.850478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11672.008849] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11673.058198] LustreError: 407495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11677.157820] LustreError: 407494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11679.035984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11679.260594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11682.577057] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11689.018044] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11693.789850] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11699.636585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11699.888779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11705.120728] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11712.062769] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11713.784507] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11716.159376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11722.720980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11722.741774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11722.749720] Lustre: Skipped 1 previous similar message [11727.449743] Lustre: server umount lustre-OST0000 complete [11732.961714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11732.963731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11732.971647] Lustre: Skipped 2 previous similar messages [11732.979734] Lustre: Skipped 1 previous similar message [11738.080712] LustreError: 408582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11738.097391] LustreError: 408582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11738.793744] Lustre: server umount lustre-MDT0000 complete [11742.183811] LustreError: 407494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11742.209204] LustreError: 407494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11742.683736] LustreError: 407475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685157 with bad export cookie 5679230046900525748 [11742.688812] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11742.692438] LustreError: 407475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11743.026278] Lustre: server umount lustre-MDT0001 complete [11753.316892] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [11756.296053] Key type lgssc unregistered [11756.650681] LNet: 411217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11756.657161] LNetError: 411217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11756.673970] LNet: Removed LNI 192.168.201.149@tcp [11757.336963] Key type .llcrypt unregistered [11757.338809] Key type ._llcrypt unregistered [11779.051337] Key type ._llcrypt registered [11779.053641] Key type .llcrypt registered [11779.158908] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [11779.454380] Key type .llcrypt unregistered [11779.457912] Key type ._llcrypt unregistered [11795.643451] Key type ._llcrypt registered [11795.647574] Key type .llcrypt registered [11795.755160] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [11796.686153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11796.727692] alg: No test for adler32 (adler32-zlib) [11797.758961] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [11797.936987] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [11799.559268] Key type lgssc registered [11800.388567] Lustre: Echo OBD driver; http://www.lustre.org/ [11810.225860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11810.273469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11812.382784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11818.659441] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11822.559963] LustreError: 413434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.599242] LustreError: 413434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11823.584806] LustreError: 413433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11828.716678] LustreError: 413434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11830.343152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11830.809929] LustreError: 413433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11830.841563] LustreError: 413433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11830.850534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11835.875577] LustreError: 413434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11835.913138] LustreError: 413434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11837.942893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11838.400543] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11848.138367] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11848.170450] LustreError: 413429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.198649] LustreError: 413429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11858.283211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11858.821148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11865.273931] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [11878.881980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11878.886788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11878.897101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11879.395778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11879.411929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11879.418970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11879.443789] Lustre: Skipped 1 previous similar message [11882.819692] Lustre: server umount lustre-OST0000 complete [11889.640867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11889.642422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11889.658793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11889.678596] Lustre: Skipped 1 previous similar message [11894.159987] Lustre: server umount lustre-MDT0000 complete [11894.766658] LustreError: 413433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11894.798177] LustreError: 413433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11898.022089] LustreError: 413412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685313 with bad export cookie 16898858849845430550 [11898.024803] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11898.029730] LustreError: 413412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11898.594595] Lustre: server umount lustre-MDT0001 complete [11910.720315] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [11913.410990] Key type lgssc unregistered [11913.710275] LNet: 416584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11913.729748] LNetError: 416584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11914.796370] LNet: Removed LNI 192.168.201.149@tcp [11915.583092] Key type .llcrypt unregistered [11915.586101] Key type ._llcrypt unregistered [11943.056533] Key type ._llcrypt registered [11943.059566] Key type .llcrypt registered [11943.190232] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [11944.088117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11944.209252] alg: No test for adler32 (adler32-zlib) [11945.233159] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [11945.441574] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [11947.103170] Key type lgssc registered [11948.000843] Lustre: Echo OBD driver; http://www.lustre.org/ [11955.219745] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11955.230471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11970.783494] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11975.905555] LustreError: 417973:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.149@tcp: failed processing log, type 4: rc = -110 [12002.591541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12008.805321] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12016.234579] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12023.062630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12023.436186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12023.515164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12027.829292] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12028.897403] LustreError: 417998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.915170] LustreError: 417998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12034.024562] LustreError: 417999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.453616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12036.868778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12041.213320] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12048.625144] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12055.505917] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12064.738798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12064.741887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12064.769361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12067.808407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12067.814543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.829043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12068.836149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12068.858042] Lustre: Skipped 1 previous similar message [12073.958449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12073.970706] Lustre: Skipped 1 previous similar message [12078.047220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12078.289560] Lustre: server umount lustre-OST0000 complete [12083.169149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12083.174043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12083.206729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12089.314198] LustreError: 419011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.329667] LustreError: 419011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12090.122398] Lustre: server umount lustre-MDT0000 complete [12093.879608] LustreError: 417980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685508 with bad export cookie 10188581606757581704 [12093.886034] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12093.892095] LustreError: 417980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12094.168875] Lustre: server umount lustre-MDT0001 complete [12104.858381] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [12107.886609] Key type lgssc unregistered [12108.166589] LNet: 421542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12108.171750] LNetError: 421542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12108.183678] LNet: Removed LNI 192.168.201.149@tcp [12108.968714] Key type .llcrypt unregistered [12108.971197] Key type ._llcrypt unregistered [12135.142554] Key type ._llcrypt registered [12135.144424] Key type .llcrypt registered [12135.281579] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [12136.713632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12136.821906] alg: No test for adler32 (adler32-zlib) [12137.812575] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [12138.097053] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [12139.759269] Key type lgssc registered [12141.223938] Lustre: Echo OBD driver; http://www.lustre.org/ [12149.036841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12149.068590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12150.428374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12154.955994] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12159.970780] LustreError: 422963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12160.018071] LustreError: 422963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12163.557476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12163.999623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12167.832629] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12174.405264] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12179.801586] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12186.581904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12187.146873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12194.805455] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12195.344715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12209.483971] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12217.764413] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [12218.149697] Lustre: server umount lustre-MDT0000 complete [12220.395213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12220.407338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12220.428651] LustreError: 422964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.645410] LustreError: 422945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.49@tcp arrived at 1789685635 with bad export cookie 4402042522700080506 [12220.670312] LustreError: 422945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12220.912168] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12220.930726] Lustre: Skipped 1 previous similar message [12223.111635] LustreError: 423698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685638 with bad export cookie 4402042522700079610 [12223.130828] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12223.134468] LustreError: 423698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12225.511340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12225.523591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12225.537213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12226.021798] LustreError: 423691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12226.036811] LustreError: 423691:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12228.064843] LustreError: 422964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12228.068571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12228.078232] LustreError: 422964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12228.094201] Lustre: Skipped 1 previous similar message [12229.418761] Lustre: server umount lustre-OST0000 complete [12233.189363] LustreError: 423691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12233.211038] LustreError: 423691:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12236.798366] Lustre: server umount lustre-MDT0001 complete [12250.092322] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [12252.784818] Key type lgssc unregistered [12253.032467] LNet: 426648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12253.043355] LNetError: 426648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12253.069541] LNet: Removed LNI 192.168.201.149@tcp [12253.782737] Key type .llcrypt unregistered [12253.785975] Key type ._llcrypt unregistered [12281.531448] Key type ._llcrypt registered [12281.536426] Key type .llcrypt registered [12281.676733] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [12282.955873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12282.993732] alg: No test for adler32 (adler32-zlib) [12284.078198] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [12284.272243] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [12285.911311] Key type lgssc registered [12287.182898] Lustre: Echo OBD driver; http://www.lustre.org/ [12294.924056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12294.942161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12296.358954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12300.828842] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12306.917590] LustreError: 428063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12306.935543] LustreError: 428063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12310.062116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12310.561419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12316.640846] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12328.278860] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12337.966198] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12348.912230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12349.359727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12358.069300] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12365.942272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12368.926513] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12376.573477] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12376.620174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12376.628575] Lustre: Skipped 1 previous similar message [12377.623046] Lustre: server umount lustre-MDT0001 complete [12381.674947] LustreError: 428063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.728513] LustreError: 428063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12386.788437] LustreError: 428792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12386.837752] LustreError: 428792:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12391.728188] Lustre: server umount lustre-OST0000 complete [12391.909131] LustreError: 428064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12397.038934] LustreError: 428792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12407.885116] Lustre: server umount lustre-MDT0000 complete [12425.988644] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [12429.959617] Key type lgssc unregistered [12430.335274] LNet: 431751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12430.350288] LNetError: 431751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12430.389691] LNet: Removed LNI 192.168.201.149@tcp [12431.554829] Key type .llcrypt unregistered [12431.557772] Key type ._llcrypt unregistered [12460.501475] Key type ._llcrypt registered [12460.504569] Key type .llcrypt registered [12460.646905] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [12476.815934] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [12478.744221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12478.862958] alg: No test for adler32 (adler32-zlib) [12480.072472] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [12480.306522] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [12481.959277] Key type lgssc registered [12483.360916] Lustre: Echo OBD driver; http://www.lustre.org/ [12532.147494] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [12551.480239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12551.529426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12552.820417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12552.852992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12552.911966] Lustre: lustre-MDT0000: new disk, initializing [12553.016307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12553.045601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12558.374269] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12575.231797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12575.608948] Lustre: lustre-OST0000: new disk, initializing [12575.616925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12575.630407] Lustre: Skipped 1 previous similar message [12575.640766] Lustre: 436808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12575.781706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12582.519972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12582.528640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12582.560747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12582.766429] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12593.057340] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12611.522959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12611.686807] Lustre: 435791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12611.849465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12612.047724] Lustre: lustre-MDT0001: new disk, initializing [12612.276756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12612.366403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12612.386124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12618.197719] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12631.013557] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12631.403375] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12637.029692] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12637.240511] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12644.833807] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12644.855504] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12644.869878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12644.881166] Lustre: Skipped 1 previous similar message [12648.390379] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [12649.974369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12653.452312] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [12653.462950] Lustre: Skipped 1 previous similar message [12658.143212] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12658.342984] Lustre: server umount lustre-MDT0001 complete [12663.690926] LustreError: 437135:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12663.727164] LustreError: 437135:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12665.314777] LustreError: 435799:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12665.343227] LustreError: 435799:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12670.442658] LustreError: 437135:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12670.469562] LustreError: 437135:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12675.553450] LustreError: 435803:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12675.574262] LustreError: 435803:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12680.673700] LustreError: 439393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12685.281670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12685.297217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12685.315549] Lustre: Skipped 1 previous similar message [12685.330271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12685.340637] Lustre: Skipped 2 previous similar messages [12686.529588] Lustre: server umount lustre-OST0000 complete [12690.919194] LustreError: 439393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12690.943363] LustreError: 439393:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12698.159087] Lustre: server umount lustre-MDT0000 complete [12730.450511] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [12734.447170] Key type lgssc unregistered [12734.823636] LNet: 440598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12734.845438] LNetError: 440598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12734.886072] LNet: Removed LNI 192.168.201.149@tcp [12736.121962] Key type .llcrypt unregistered [12736.125060] Key type ._llcrypt unregistered [12792.886710] Key type ._llcrypt registered [12792.891246] Key type .llcrypt registered [12793.037647] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [12793.375136] Key type .llcrypt unregistered [12793.377443] Key type ._llcrypt unregistered [12813.142939] Key type ._llcrypt registered [12813.146208] Key type .llcrypt registered [12813.284096] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [12814.445906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12814.501084] alg: No test for adler32 (adler32-zlib) [12815.867540] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [12816.189202] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [12817.847375] Key type lgssc registered [12819.028175] Lustre: Echo OBD driver; http://www.lustre.org/ [12849.541297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12849.585926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12850.823166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12850.893278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12850.974881] Lustre: lustre-MDT0000: new disk, initializing [12851.095498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12851.118243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12855.407830] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12866.983766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12867.196831] Lustre: 443971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12867.268934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12867.276773] Lustre: Skipped 1 previous similar message [12867.361560] Lustre: lustre-MDT0001: new disk, initializing [12867.491908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12867.566981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12867.583701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12872.188933] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12882.660613] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12888.699422] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12896.311355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12896.773335] Lustre: lustre-OST0000: new disk, initializing [12896.785201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12896.792283] Lustre: 445940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12896.929160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12902.989398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12903.001766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12903.086531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12903.583271] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12916.861845] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12928.992515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12928.999565] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12929.004314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12929.043340] Lustre: Skipped 2 previous similar messages [12931.652140] Lustre: server umount lustre-MDT0000 complete [12936.231621] LustreError: 445950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686351 with bad export cookie 5067585612996665897 [12936.236680] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12936.253409] LustreError: 445950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12936.557456] Lustre: server umount lustre-MDT0001 complete [12952.057651] Lustre: server umount lustre-OST0000 complete [12964.572483] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:06:18 (1789686378) [12974.818498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12975.653734] LustreError: 447971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.678482] LustreError: 447971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12975.753363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12980.620713] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [12981.220638] LustreError: 447972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12985.319384] LustreError: 447971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12990.447694] LustreError: 447972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12991.363811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12992.067563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12998.372710] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [13007.976203] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13015.631269] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13036.482332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13036.762169] Lustre: Found index 0 for lustre-OST0000, updating log [13037.039854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13044.933319] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [13046.333979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13054.838380] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13061.622564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13061.637413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13061.656067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13061.662178] Lustre: Skipped 2 previous similar messages [13062.698994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13064.818037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13064.833755] Lustre: Skipped 1 previous similar message [13068.010399] Lustre: server umount lustre-OST0000 complete [13084.653286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13084.684684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13084.694676] Lustre: Skipped 2 previous similar messages [13089.249372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13089.255644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13091.087743] Lustre: server umount lustre-MDT0000 complete [13093.861616] LustreError: 447971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13093.876549] LustreError: 447971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13095.394395] LustreError: 447953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686510 with bad export cookie 5067585612996667283 [13095.403181] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13095.413262] LustreError: 447953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13095.752679] Lustre: server umount lustre-MDT0001 complete [13105.184162] LNet: 1 local NIs in recovery (showing 1): 192.168.201.149@tcp [13105.244236] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:08:39 (1789686519) [13704.653848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13704.894306] LustreError: 454381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13704.904619] LustreError: 454381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13704.948632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13707.057916] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [13711.078356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13711.244957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13713.350376] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [13717.189617] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13720.059749] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13736.741145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13736.858811] Lustre: Found index 0 for lustre-OST0000, updating log [13737.002386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13739.718473] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [13743.571284] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13749.217102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13749.230125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13749.236745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13749.241756] Lustre: Skipped 2 previous similar messages [13751.071831] Lustre: server umount lustre-OST0000 complete [13755.878261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13755.885805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13755.903771] Lustre: Skipped 1 previous similar message [13755.910720] Lustre: Skipped 1 previous similar message [13757.471419] LNet: 1 local NIs in recovery (showing 1): 192.168.201.149@tcp [13757.985563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13757.992106] Lustre: Skipped 3 previous similar messages [13761.678201] Lustre: server umount lustre-MDT0000 complete [13762.092259] LustreError: 454375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13762.124139] LustreError: 454375:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13767.393160] LustreError: 454363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687182 with bad export cookie 5067585612996668361 [13767.404041] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13767.427562] LustreError: 454363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13768.560218] Lustre: server umount lustre-MDT0001 complete [13817.342929] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [13830.498701] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [13882.204785] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:21:36 (1789687296) [13901.812996] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [13904.870376] Key type lgssc unregistered [13905.186587] LNet: 460996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13905.200818] LNetError: 460996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13997.481189] LNet: Removed LNI 192.168.201.149@tcp [13998.257217] Key type .llcrypt unregistered [13998.259433] Key type ._llcrypt unregistered [14016.419424] Key type ._llcrypt registered [14016.423130] Key type .llcrypt registered [14016.553758] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [14017.719678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14017.803767] alg: No test for adler32 (adler32-zlib) [14018.869527] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [14019.080413] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [14020.767674] Key type lgssc registered [14022.244946] Lustre: Echo OBD driver; http://www.lustre.org/ [14027.385282] LNet: Added LNI 192.168.201.149@tcp100 [8/256/0/180] [14028.225387] LNet: Added LNI 192.168.201.149@tcp101 [8/256/0/180] [14028.962277] LNet: Added LNI 192.168.201.149@tcp102 [8/256/0/180] [14029.674198] LNet: Added LNI 192.168.201.149@tcp103 [8/256/0/180] [14030.593630] LNet: Added LNI 192.168.201.149@tcp104 [8/256/0/180] [14031.700220] LNet: Added LNI 192.168.201.149@tcp105 [8/256/0/180] [14032.898361] LNet: Added LNI 192.168.201.149@tcp106 [8/256/0/180] [14033.907502] LNet: Added LNI 192.168.201.149@tcp107 [8/256/0/180] [14034.803969] LNet: Added LNI 192.168.201.149@tcp108 [8/256/0/180] [14035.821651] LNet: Added LNI 192.168.201.149@tcp109 [8/256/0/180] [14036.986717] LNet: Added LNI 192.168.201.149@tcp110 [8/256/0/180] [14037.913354] LNet: Added LNI 192.168.201.149@tcp111 [8/256/0/180] [14039.033213] LNet: Added LNI 192.168.201.149@tcp112 [8/256/0/180] [14040.308775] LNet: Added LNI 192.168.201.149@tcp113 [8/256/0/180] [14041.425848] LNet: Added LNI 192.168.201.149@tcp114 [8/256/0/180] [14042.610549] LNet: Added LNI 192.168.201.149@tcp115 [8/256/0/180] [14043.928854] LNet: Added LNI 192.168.201.149@tcp116 [8/256/0/180] [14044.901722] LNet: Added LNI 192.168.201.149@tcp117 [8/256/0/180] [14046.085346] LNet: Added LNI 192.168.201.149@tcp118 [8/256/0/180] [14046.981107] LNet: Added LNI 192.168.201.149@tcp119 [8/256/0/180] [14047.997711] LNet: Added LNI 192.168.201.149@tcp120 [8/256/0/180] [14048.818950] LNet: Added LNI 192.168.201.149@tcp121 [8/256/0/180] [14050.129986] LNet: Added LNI 192.168.201.149@tcp122 [8/256/0/180] [14050.945214] LNet: Added LNI 192.168.201.149@tcp123 [8/256/0/180] [14051.602291] LNet: Added LNI 192.168.201.149@tcp124 [8/256/0/180] [14052.439191] LNet: Added LNI 192.168.201.149@tcp125 [8/256/0/180] [14053.318285] LNet: Added LNI 192.168.201.149@tcp126 [8/256/0/180] [14054.554465] LNet: Added LNI 192.168.201.149@tcp127 [8/256/0/180] [14055.764855] LNet: Added LNI 192.168.201.149@tcp128 [8/256/0/180] [14056.607308] LNet: Added LNI 192.168.201.149@tcp129 [8/256/0/180] [14057.771324] LNet: Added LNI 192.168.201.149@tcp130 [8/256/0/180] [14058.711630] LNet: Added LNI 192.168.201.149@tcp131 [8/256/0/180] [14060.330522] LNet: Added LNI 192.168.201.149@tcp132 [8/256/0/180] [14061.541040] LNet: Added LNI 192.168.201.149@tcp133 [8/256/0/180] [14062.644869] LNet: Added LNI 192.168.201.149@tcp134 [8/256/0/180] [14063.915525] LNet: Added LNI 192.168.201.149@tcp135 [8/256/0/180] [14071.445626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14071.480546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14072.693582] Lustre: 464023:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [14072.840776] LNetError: 461478:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.149 [14072.862067] LNetError: Protocol error connecting to 192.168.201.149@tcp122 at host 192.168.201.149:988: is it running a compatible version of Lustre? [14072.872310] LNetError: Skipped 1 previous similar message [14072.931203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14072.955765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14073.083692] Lustre: lustre-MDT0000: new disk, initializing [14073.211966] Lustre: 464080:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [14073.224669] Lustre: 464080:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14073.244784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14073.269546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14073.378473] LNet: 2 local NIs in recovery (showing 2): 192.168.201.149@tcp120, 192.168.201.149@tcp122 [14073.386411] LNetError: 464054:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.149 [14073.421847] LNetError: 464054:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14073.434542] LNetError: Protocol error connecting to 192.168.201.149@tcp103 at host 192.168.201.149:988: is it running a compatible version of Lustre? [14073.466388] LNetError: Skipped 9 previous similar messages [14075.427261] LNetError: 461479:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.149 [14075.451440] LNetError: 461479:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14075.463743] LNetError: Protocol error connecting to 192.168.201.149@tcp120 at host 192.168.201.149:988: is it running a compatible version of Lustre? [14079.525891] LNetError: 464055:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.149 [14079.550507] LNetError: Protocol error connecting to 192.168.201.149@tcp120 at host 192.168.201.149:988: is it running a compatible version of Lustre? [14079.855222] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [14087.713817] LNetError: 461478:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.149 [14087.733492] LNetError: Protocol error connecting to 192.168.201.149@tcp120 at host 192.168.201.149:988: is it running a compatible version of Lustre? [14091.375923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14091.499917] Lustre: 464038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14091.549426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14091.553146] Lustre: Skipped 1 previous similar message [14091.610652] Lustre: lustre-MDT0001: new disk, initializing [14091.643410] Lustre: 464842:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [14091.650889] Lustre: 464842:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14091.760653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14091.796683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14091.809514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14097.129488] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [14100.041429] Lustre: 465205:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [14100.057856] Lustre: 465205:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14102.061618] LNetError: 464058:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.149 [14102.087165] LNetError: Protocol error connecting to 192.168.201.149@tcp123 at host 192.168.201.149:988: is it running a compatible version of Lustre? [14106.940791] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14112.959290] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14120.484292] LNetError: 464045:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.149 [14120.516351] LNetError: 464045:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14120.532599] LNetError: Protocol error connecting to 192.168.201.149@tcp103 at host 192.168.201.149:988: is it running a compatible version of Lustre? [14120.570971] LNetError: Skipped 11 previous similar messages [14121.237713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14121.582107] Lustre: lustre-OST0000: new disk, initializing [14121.585637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14121.596613] Lustre: 466015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14121.698173] Lustre: 466020:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [14121.710595] Lustre: 466020:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14121.783482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14127.697072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14127.708202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14127.897216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14129.073241] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [14133.791514] LNet: 1 local NIs in recovery (showing 1): 192.168.201.149@tcp120 [14139.107645] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14148.580731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14148.582810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14148.598134] LustreError: Skipped 1 previous similar message [14148.599904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14148.654566] Lustre: Skipped 1 previous similar message [14152.662252] Lustre: server umount lustre-OST0000 complete [14153.262451] LNetError: 464049:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.149 [14153.284331] LNetError: 464049:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [14153.298987] LNetError: Protocol error connecting to 192.168.201.149@tcp123 at host 192.168.201.149:988: is it running a compatible version of Lustre? [14153.311708] LNetError: Skipped 14 previous similar messages [14158.821786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14158.863807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14158.877292] Lustre: Skipped 1 previous similar message [14163.427570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14163.448832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14163.477037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14165.214219] Lustre: server umount lustre-MDT0000 complete [14170.181115] LustreError: 464033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687585 with bad export cookie 3822551616877456340 [14170.194888] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14170.467788] Lustre: server umount lustre-MDT0001 complete [14181.876744] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [14185.277925] Key type lgssc unregistered [14185.619087] LNet: 467931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14185.639475] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14185.648910] LNet: Removed LNI 192.168.201.149@tcp [14186.657929] LNet: Removed LNI 192.168.201.149@tcp100 [14186.660163] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14186.680455] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14186.684064] LNet: Removed LNI 192.168.201.149@tcp101 [14187.743864] LNet: Removed LNI 192.168.201.149@tcp102 [14187.766303] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14187.787324] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14187.811705] LNet: Removed LNI 192.168.201.149@tcp103 [14188.832754] LNet: Removed LNI 192.168.201.149@tcp104 [14189.855159] LNet: Removed LNI 192.168.201.149@tcp105 [14189.862458] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14189.883399] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14189.898464] LNet: Removed LNI 192.168.201.149@tcp106 [14190.944851] LNet: Removed LNI 192.168.201.149@tcp107 [14191.967164] LNet: Removed LNI 192.168.201.149@tcp108 [14192.992460] LNet: Removed LNI 192.168.201.149@tcp109 [14194.015291] LNet: Removed LNI 192.168.201.149@tcp110 [14194.030223] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14194.048236] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14194.059721] LNet: Removed LNI 192.168.201.149@tcp111 [14195.105674] LNet: Removed LNI 192.168.201.149@tcp112 [14196.127201] LNet: Removed LNI 192.168.201.149@tcp113 [14197.151731] LNet: Removed LNI 192.168.201.149@tcp114 [14198.175493] LNet: Removed LNI 192.168.201.149@tcp115 [14199.199850] LNet: Removed LNI 192.168.201.149@tcp116 [14200.223325] LNet: Removed LNI 192.168.201.149@tcp117 [14201.312099] LNet: Removed LNI 192.168.201.149@tcp118 [14202.335439] LNet: Removed LNI 192.168.201.149@tcp119 [14202.340636] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14202.358759] LNetError: 467931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14202.381250] LNet: Removed LNI 192.168.201.149@tcp120 [14203.424043] LNet: Removed LNI 192.168.201.149@tcp121 [14204.447165] LNet: Removed LNI 192.168.201.149@tcp122 [14205.471161] LNet: Removed LNI 192.168.201.149@tcp123 [14206.497182] LNet: Removed LNI 192.168.201.149@tcp124 [14207.519170] LNet: Removed LNI 192.168.201.149@tcp125 [14208.543537] LNet: Removed LNI 192.168.201.149@tcp126 [14209.567897] LNet: Removed LNI 192.168.201.149@tcp127 [14210.591384] LNet: Removed LNI 192.168.201.149@tcp128 [14211.615176] LNet: Removed LNI 192.168.201.149@tcp129 [14211.618600] LNet: Removed LNI 192.168.201.149@tcp130 [14211.621203] LNet: Removed LNI 192.168.201.149@tcp131 [14211.626614] LNet: Removed LNI 192.168.201.149@tcp132 [14211.629539] LNet: Removed LNI 192.168.201.149@tcp133 [14211.633459] LNet: Removed LNI 192.168.201.149@tcp134 [14211.643996] LNet: Removed LNI 192.168.201.149@tcp135 [14212.505099] Key type .llcrypt unregistered [14212.508478] Key type ._llcrypt unregistered [14238.248798] Key type ._llcrypt registered [14238.250575] Key type .llcrypt registered [14238.379509] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [14254.855767] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [14256.209951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14256.405248] alg: No test for adler32 (adler32-zlib) [14257.539359] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [14257.846700] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [14259.535225] Key type lgssc registered [14261.074698] Lustre: Echo OBD driver; http://www.lustre.org/ [14297.492863] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:28:31 (1789687711) [14316.941476] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [14320.168971] Key type lgssc unregistered [14320.675558] LNet: 472016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14320.684190] LNetError: 472016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14320.707498] LNet: Removed LNI 192.168.201.149@tcp [14321.579706] Key type .llcrypt unregistered [14321.586982] Key type ._llcrypt unregistered [14340.880749] Key type ._llcrypt registered [14340.882771] Key type .llcrypt registered [14340.958811] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [14342.174732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14342.247903] alg: No test for adler32 (adler32-zlib) [14343.415961] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [14343.659422] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [14345.335230] Key type lgssc registered [14346.427858] Lustre: Echo OBD driver; http://www.lustre.org/ [14356.633505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14356.653097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14357.974197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14358.014265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14358.149759] Lustre: lustre-MDT0000: new disk, initializing [14358.371335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14358.402942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14363.985867] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [14376.009289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14376.153704] Lustre: 473229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14376.215220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14376.219355] Lustre: Skipped 1 previous similar message [14376.447926] Lustre: lustre-MDT0001: new disk, initializing [14376.572418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14376.645533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14376.670374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14381.781938] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [14391.878619] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14398.055730] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14404.839119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14405.011667] Lustre: lustre-OST0000: new disk, initializing [14405.015159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14405.020405] Lustre: 475196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14405.080686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14411.966558] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [14413.373883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14413.391847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14413.520299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14422.467230] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14427.483388] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing lnet_if_list [14433.903916] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14433.942429] LNet: Added LNI 192.168.201.149@tcp10 [8/256/0/180] [14440.616454] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14440.677185] LNet: Added LNI 192.168.201.149@tcp11 [8/256/0/180] [14446.476127] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14446.532829] LNet: Added LNI 192.168.201.149@tcp12 [8/256/0/180] [14451.801726] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14451.835864] LNet: Added LNI 192.168.201.149@tcp13 [8/256/0/180] [14456.965960] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14456.991033] LNet: Added LNI 192.168.201.149@tcp14 [8/256/0/180] [14462.272223] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14462.318304] LNet: Added LNI 192.168.201.149@tcp15 [8/256/0/180] [14468.167276] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14468.220781] LNet: Added LNI 192.168.201.149@tcp16 [8/256/0/180] [14474.253557] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14474.317928] LNet: Added LNI 192.168.201.149@tcp17 [8/256/0/180] [14479.743737] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14479.765350] LNet: Added LNI 192.168.201.149@tcp18 [8/256/0/180] [14485.442941] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14485.504738] LNet: Added LNI 192.168.201.149@tcp19 [8/256/0/180] [14491.969595] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14492.010869] LNet: Added LNI 192.168.201.149@tcp20 [8/256/0/180] [14497.485650] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14497.539424] LNet: Added LNI 192.168.201.149@tcp21 [8/256/0/180] [14503.817474] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14503.885409] LNet: Added LNI 192.168.201.149@tcp22 [8/256/0/180] [14510.062876] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14510.091617] LNet: Added LNI 192.168.201.149@tcp23 [8/256/0/180] [14517.058108] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14517.088676] LNet: Added LNI 192.168.201.149@tcp24 [8/256/0/180] [14522.885040] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14522.922420] LNet: Added LNI 192.168.201.149@tcp25 [8/256/0/180] [14529.126820] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14529.144835] LNet: Added LNI 192.168.201.149@tcp26 [8/256/0/180] [14536.100283] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14536.143103] LNet: Added LNI 192.168.201.149@tcp27 [8/256/0/180] [14543.489120] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14543.528891] LNet: Added LNI 192.168.201.149@tcp28 [8/256/0/180] [14550.275398] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14550.318730] LNet: Added LNI 192.168.201.149@tcp29 [8/256/0/180] [14557.653916] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14557.693798] LNet: Added LNI 192.168.201.149@tcp30 [8/256/0/180] [14565.019519] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14565.043150] LNet: Added LNI 192.168.201.149@tcp31 [8/256/0/180] [14573.341189] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14573.399594] LNet: Added LNI 192.168.201.149@tcp32 [8/256/0/180] [14581.628345] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14581.677931] LNet: Added LNI 192.168.201.149@tcp33 [8/256/0/180] [14589.524670] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14589.551180] LNet: Added LNI 192.168.201.149@tcp34 [8/256/0/180] [14597.526467] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14597.555830] LNet: Added LNI 192.168.201.149@tcp35 [8/256/0/180] [14607.392334] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14607.458843] LNet: Added LNI 192.168.201.149@tcp36 [8/256/0/180] [14617.869150] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14617.925448] LNet: Added LNI 192.168.201.149@tcp37 [8/256/0/180] [14628.690599] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14628.752844] LNet: Added LNI 192.168.201.149@tcp38 [8/256/0/180] [14638.412749] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14638.444722] LNet: Added LNI 192.168.201.149@tcp39 [8/256/0/180] [14648.437610] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14648.488521] LNet: Added LNI 192.168.201.149@tcp40 [8/256/0/180] [14658.059284] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14658.095968] LNet: Added LNI 192.168.201.149@tcp41 [8/256/0/180] [14667.128649] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14667.150346] LNet: Added LNI 192.168.201.149@tcp42 [8/256/0/180] [14674.132781] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14674.160865] LNet: Added LNI 192.168.201.149@tcp43 [8/256/0/180] [14681.940513] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14681.964389] LNet: Added LNI 192.168.201.149@tcp44 [8/256/0/180] [14690.957380] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14691.007688] LNet: Added LNI 192.168.201.149@tcp45 [8/256/0/180] [14699.820365] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14699.849456] LNet: Added LNI 192.168.201.149@tcp46 [8/256/0/180] [14709.377649] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14709.400317] LNet: Added LNI 192.168.201.149@tcp47 [8/256/0/180] [14718.722551] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14718.748445] LNet: Added LNI 192.168.201.149@tcp48 [8/256/0/180] [14728.253975] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14728.275633] LNet: Added LNI 192.168.201.149@tcp49 [8/256/0/180] [14736.070720] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14736.093047] LNet: Added LNI 192.168.201.149@tcp50 [8/256/0/180] [14755.297811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14755.309243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14755.322706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14756.836371] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14756.837361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14756.864364] Lustre: Skipped 1 previous similar message [14756.894703] Lustre: Skipped 2 previous similar messages [14760.590031] Lustre: server umount lustre-MDT0000 complete [14765.059787] LustreError: 473222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688180 with bad export cookie 13283639745249415251 [14765.067986] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14765.075732] LustreError: 473222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14765.602414] Lustre: server umount lustre-MDT0001 complete [14783.458181] Lustre: 472500:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789688182/real 1789688182] req@000000002c2968e3 x1876623630713472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789688198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14783.503350] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14786.597308] Lustre: 472500:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789688185/real 1789688185] req@00000000508c3566 x1876623630713600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789688201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14786.889922] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [14788.611687] Lustre: 472501:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789688187/real 1789688187] req@00000000feca5bcb x1876623630713728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789688203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14791.712139] Lustre: 472501:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789688190/real 1789688190] req@000000007f372af5 x1876623630713984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789688206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14796.774557] Lustre: 472500:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789688195/real 1789688195] req@000000005b04ebce x1876623630714112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789688211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14797.083438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14810.417718] LustreError: 475189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14810.447647] LustreError: 475189:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14810.482152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14812.522393] LustreError: 475190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14812.545298] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14813.545469] LustreError: 475189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14815.259627] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [14818.786660] LustreError: 475190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14823.905600] LustreError: 475189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14824.442898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14824.717168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14828.869395] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14829.064808] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [14837.525635] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14843.964212] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14865.889251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14865.900553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14865.915691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14866.913976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14869.984799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14869.995864] Lustre: Skipped 2 previous similar messages [14875.119632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14875.138638] Lustre: Skipped 1 previous similar message [14879.199618] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14879.485422] Lustre: server umount lustre-OST0000 complete [14885.345139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14885.352672] Lustre: Skipped 1 previous similar message [14885.359651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14885.365466] Lustre: Skipped 1 previous similar message [14890.029614] Lustre: server umount lustre-MDT0000 complete [14890.465526] LustreError: 499465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14890.492467] LustreError: 499465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14894.538211] LustreError: 473222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688309 with bad export cookie 13283639745249417820 [14894.543094] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14894.550192] LustreError: 473222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14894.937616] Lustre: server umount lustre-MDT0001 complete [14907.763956] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [14911.149188] Key type lgssc unregistered [14911.638091] LNet: 502262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14911.651514] LNetError: 502262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14911.657374] LNet: Removed LNI 192.168.201.149@tcp [14911.659815] LNet: Removed LNI 192.168.201.149@tcp10 [14911.662703] LNet: Removed LNI 192.168.201.149@tcp11 [14911.665870] LNet: Removed LNI 192.168.201.149@tcp12 [14911.669371] LNet: Removed LNI 192.168.201.149@tcp13 [14911.673577] LNet: Removed LNI 192.168.201.149@tcp14 [14911.676847] LNet: Removed LNI 192.168.201.149@tcp15 [14911.679915] LNet: Removed LNI 192.168.201.149@tcp16 [14911.684590] LNet: Removed LNI 192.168.201.149@tcp17 [14911.688670] LNet: Removed LNI 192.168.201.149@tcp18 [14911.690440] LNet: Removed LNI 192.168.201.149@tcp19 [14911.694241] LNet: Removed LNI 192.168.201.149@tcp20 [14911.697329] LNet: Removed LNI 192.168.201.149@tcp21 [14911.700572] LNet: Removed LNI 192.168.201.149@tcp22 [14911.703597] LNet: Removed LNI 192.168.201.149@tcp23 [14911.706323] LNet: Removed LNI 192.168.201.149@tcp24 [14911.708337] LNet: Removed LNI 192.168.201.149@tcp25 [14911.711303] LNet: Removed LNI 192.168.201.149@tcp26 [14911.714382] LNet: Removed LNI 192.168.201.149@tcp27 [14911.719704] LNet: Removed LNI 192.168.201.149@tcp28 [14911.724777] LNet: Removed LNI 192.168.201.149@tcp29 [14911.728700] LNet: Removed LNI 192.168.201.149@tcp30 [14911.732111] LNet: Removed LNI 192.168.201.149@tcp31 [14911.733882] LNet: Removed LNI 192.168.201.149@tcp32 [14911.736160] LNet: Removed LNI 192.168.201.149@tcp33 [14911.737871] LNet: Removed LNI 192.168.201.149@tcp34 [14911.739844] LNet: Removed LNI 192.168.201.149@tcp35 [14911.742190] LNet: Removed LNI 192.168.201.149@tcp36 [14911.747375] LNet: Removed LNI 192.168.201.149@tcp37 [14911.750915] LNet: Removed LNI 192.168.201.149@tcp38 [14911.753382] LNet: Removed LNI 192.168.201.149@tcp39 [14911.755115] LNet: Removed LNI 192.168.201.149@tcp40 [14911.756531] LNet: Removed LNI 192.168.201.149@tcp41 [14911.758777] LNet: Removed LNI 192.168.201.149@tcp42 [14911.761393] LNet: Removed LNI 192.168.201.149@tcp43 [14911.763807] LNet: Removed LNI 192.168.201.149@tcp44 [14911.766321] LNet: Removed LNI 192.168.201.149@tcp45 [14911.768716] LNet: Removed LNI 192.168.201.149@tcp46 [14911.771297] LNet: Removed LNI 192.168.201.149@tcp47 [14911.773937] LNet: Removed LNI 192.168.201.149@tcp48 [14911.776916] LNet: Removed LNI 192.168.201.149@tcp49 [14911.787184] LNet: Removed LNI 192.168.201.149@tcp50 [14912.591489] Key type .llcrypt unregistered [14912.593172] Key type ._llcrypt unregistered [14935.519253] Key type ._llcrypt registered [14935.523524] Key type .llcrypt registered [14935.671373] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [14951.062270] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [14952.144961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14952.181564] alg: No test for adler32 (adler32-zlib) [14953.213322] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [14953.418206] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [14955.071243] Key type lgssc registered [14956.227839] Lustre: Echo OBD driver; http://www.lustre.org/ [14994.199253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14994.246586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14995.568468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14995.598411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14995.669060] Lustre: lustre-MDT0000: new disk, initializing [14995.746302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14995.766846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15000.470300] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15012.139350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15012.266145] Lustre: 505457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15012.335281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15012.341210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15012.341398] Lustre: Skipped 1 previous similar message [15012.412238] Lustre: lustre-MDT0001: new disk, initializing [15012.486972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15012.528537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15012.544211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15016.763347] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15026.912213] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15033.377405] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15043.659321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15043.930227] Lustre: *** cfs_fail_loc=90f, val=0*** [15044.057813] Lustre: lustre-OST0000: new disk, initializing [15044.065765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15044.083861] Lustre: 507475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15044.186186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15049.784535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15049.799734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15049.981971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15050.534947] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15060.284223] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15070.694537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15070.701670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15070.703834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15070.709650] Lustre: Skipped 1 previous similar message [15073.148197] Lustre: server umount lustre-OST0000 complete [15079.397840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15079.403989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15079.425937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15079.438543] Lustre: Skipped 1 previous similar message [15080.931496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15083.622072] Lustre: server umount lustre-MDT0000 complete [15086.924518] LustreError: 505450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688502 with bad export cookie 3684320702190795144 [15086.935321] LustreError: 505450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15086.937492] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15087.199700] Lustre: server umount lustre-MDT0001 complete [15097.393641] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [15099.936515] Key type lgssc unregistered [15100.163195] LNet: 509378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15100.168803] LNetError: 509378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15100.181158] LNet: Removed LNI 192.168.201.149@tcp [15101.166335] Key type .llcrypt unregistered [15101.169051] Key type ._llcrypt unregistered [15121.040468] Key type ._llcrypt registered [15121.049215] Key type .llcrypt registered [15121.134313] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [15122.096021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15122.279808] alg: No test for adler32 (adler32-zlib) [15123.246856] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [15123.431459] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [15125.135259] Key type lgssc registered [15126.306859] Lustre: Echo OBD driver; http://www.lustre.org/ [15133.655268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15133.702940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15135.348218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15139.876666] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15144.933518] LustreError: 510662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15144.956523] LustreError: 510662:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15148.900777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15149.448270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15153.926432] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15163.256155] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15170.320953] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15177.600715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15178.070352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15183.987286] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15192.267885] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15205.167581] loop0: detected capacity change from 0 to 204800000 [15205.377206] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15205.405925] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15205.422590] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15211.843457] loop0: detected capacity change from 0 to 204800000 [15212.002601] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15212.013806] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15212.024927] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15218.808385] loop0: detected capacity change from 0 to 204800000 [15225.012789] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15225.157406] Lustre: fs15246-OST0000: new disk, initializing [15225.161946] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15225.169599] Lustre: 514253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15225.219601] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15231.644363] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15241.991710] loop1: detected capacity change from 0 to 204800000 [15248.078951] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15248.221864] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15248.309438] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15248.414321] Lustre: fs15246-MDT0000: new disk, initializing [15248.602410] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15248.628796] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15248.757969] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15249.836630] Lustre: 515376:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000133f3d8b x1876624448471296/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789688680 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15253.798037] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15258.927226] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15258.934334] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15258.958816] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15277.567354] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15277.583927] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15282.657548] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15283.054615] Lustre: server umount fs15246-MDT0000 complete [15287.788982] LustreError: 516204:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15292.914345] LustreError: 513102:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15293.615471] Lustre: server umount fs15246-OST0000 complete [15307.237906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15307.249148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15307.269025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15308.258526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15308.269603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15312.138541] Lustre: server umount lustre-OST0000 complete [15318.496906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15318.502635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15318.512842] Lustre: Skipped 1 previous similar message [15318.514743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15318.543887] Lustre: Skipped 3 previous similar messages [15323.070727] Lustre: server umount lustre-MDT0000 complete [15323.621829] LustreError: 510656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15323.659683] LustreError: 510656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15327.262617] LustreError: 515370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688742 with bad export cookie 9749237882719366295 [15327.264825] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15327.278574] LustreError: 515370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15327.680576] Lustre: server umount lustre-MDT0001 complete [15340.916726] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [15343.668755] Key type lgssc unregistered [15343.929627] LNet: 518476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15343.936093] LNetError: 518476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15343.948795] LNet: Removed LNI 192.168.201.149@tcp [15344.923306] Key type .llcrypt unregistered [15344.928022] Key type ._llcrypt unregistered [15365.650459] Key type ._llcrypt registered [15365.652478] Key type .llcrypt registered [15365.733533] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [15367.445194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15367.486945] alg: No test for adler32 (adler32-zlib) [15368.494743] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [15368.707361] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [15370.368429] Key type lgssc registered [15371.737171] Lustre: Echo OBD driver; http://www.lustre.org/ [15408.973632] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [15421.419877] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [15459.204742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15459.219402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15460.557327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15460.589387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15460.678503] Lustre: lustre-MDT0000: new disk, initializing [15460.771760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15460.791137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15465.415490] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15475.909605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15476.087518] Lustre: 522939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15476.162306] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15476.172115] Lustre: Skipped 1 previous similar message [15476.357930] Lustre: lustre-MDT0001: new disk, initializing [15476.493843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15476.543784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15476.566577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15480.597768] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15489.595440] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15494.746486] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15501.551340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15501.833884] Lustre: lustre-OST0000: new disk, initializing [15501.838459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15501.844882] Lustre: 524907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15501.947474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15508.325311] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [15510.573609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15510.583951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15510.651577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15517.550414] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15521.610261] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15521.811724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15526.294259] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15526.514845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15530.978685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15531.006067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15531.030419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15531.506981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15531.510538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15534.700159] Lustre: server umount lustre-OST0000 complete [15541.737309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15541.748335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15541.752256] Lustre: Skipped 1 previous similar message [15541.765935] Lustre: Skipped 2 previous similar messages [15544.645719] Lustre: server umount lustre-MDT0000 complete [15548.601396] LustreError: 522930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688963 with bad export cookie 9658406565660229431 [15548.612429] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15548.617150] LustreError: 522930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15548.975407] Lustre: server umount lustre-MDT0001 complete [15574.506462] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [15586.687690] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [15617.883095] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15339 sec ========= 19:50:32 (1789689032) [15619.522720] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:50:34 (1789689034) === [15623.635556] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:50:37 (1789689037) === [15668.328482] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [15672.453512] Key type lgssc unregistered [15673.065611] LNet: 531554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15673.082597] LNetError: 531554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15674.089968] LNet: Removed LNI 192.168.201.149@tcp [15675.488155] Key type .llcrypt unregistered [15675.496321] Key type ._llcrypt unregistered