[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 762053623 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003367] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008604] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009041] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.012063] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014088] *** VALIDATE selinux *** [ 0.023404] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028404] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031141] *** VALIDATE tmpfs *** [ 0.032471] *** VALIDATE proc *** [ 0.033262] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037023] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039006] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.044064] debug: unmapping init [mem 0xffffffffa5a59000-0xffffffffa5a60fff] [ 0.047431] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048815] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049022] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056286] rcu: Hierarchical SRCU implementation. [ 0.058604] smp: Bringing up secondary CPUs ... [ 0.059687] x86: Booting SMP configuration: [ 0.060028] .... node #0, CPUs: #1 #2 #3 [ 0.071190] smp: Brought up 1 node, 4 CPUs [ 0.073028] smpboot: Max logical packages: 1 [ 0.074014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.103024] node 0 deferred pages initialised in 27ms [ 0.107068] devtmpfs: initialized [ 0.108303] x86/mm: Memory block size: 128MB [ 0.111165] gcov: version magic: 0x41383552 [ 0.113321] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.114090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.115307] pinctrl core: initialized pinctrl subsystem [ 0.116355] [ 0.116965] ************************************************************* [ 0.117013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.118016] ** ** [ 0.119014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.120012] ** ** [ 0.121023] ** This means that this kernel is built to expose internal ** [ 0.122014] ** IOMMU data structures, which may compromise security on ** [ 0.123013] ** your system. ** [ 0.124014] ** ** [ 0.125014] ** If you see this message and you are not debugging the ** [ 0.126016] ** kernel, report this immediately to your vendor! ** [ 0.127015] ** ** [ 0.128015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129013] ************************************************************* [ 0.130732] NET: Registered protocol family 16 [ 0.131440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.132072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.133074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.134765] cpuidle: using governor menu [ 0.136692] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.139585] PCI: Using configuration type 1 for base access [ 0.142128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.152208] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.153000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.154112] cryptd: max_cpu_qlen set to 1000 [ 0.156342] ACPI: Added _OSI(Module Device) [ 0.157016] ACPI: Added _OSI(Processor Device) [ 0.158012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.159012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.163663] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.168083] ACPI: Interpreter enabled [ 0.169060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.170014] ACPI: Using IOAPIC for interrupt routing [ 0.171099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.172482] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.183349] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.184138] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.185018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.186098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.188735] acpiphp: Slot [2] registered [ 0.189112] acpiphp: Slot [5] registered [ 0.190111] acpiphp: Slot [6] registered [ 0.191143] acpiphp: Slot [7] registered [ 0.192102] acpiphp: Slot [8] registered [ 0.193122] acpiphp: Slot [9] registered [ 0.194186] acpiphp: Slot [10] registered [ 0.195267] acpiphp: Slot [3] registered [ 0.196102] acpiphp: Slot [4] registered [ 0.197228] acpiphp: Slot [11] registered [ 0.198128] acpiphp: Slot [12] registered [ 0.199121] acpiphp: Slot [13] registered [ 0.200097] acpiphp: Slot [14] registered [ 0.201108] acpiphp: Slot [15] registered [ 0.202089] acpiphp: Slot [16] registered [ 0.203119] acpiphp: Slot [17] registered [ 0.204333] acpiphp: Slot [18] registered [ 0.205099] acpiphp: Slot [19] registered [ 0.206100] acpiphp: Slot [20] registered [ 0.207272] acpiphp: Slot [21] registered [ 0.208100] acpiphp: Slot [22] registered [ 0.209121] acpiphp: Slot [23] registered [ 0.210111] acpiphp: Slot [24] registered [ 0.211122] acpiphp: Slot [25] registered [ 0.212131] acpiphp: Slot [26] registered [ 0.213197] acpiphp: Slot [27] registered [ 0.214111] acpiphp: Slot [28] registered [ 0.215104] acpiphp: Slot [29] registered [ 0.216283] acpiphp: Slot [30] registered [ 0.217149] acpiphp: Slot [31] registered [ 0.218069] PCI host bridge to bus 0000:00 [ 0.219024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.220025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.221096] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.222031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.223022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.224025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.225173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.227232] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.229502] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.235895] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.238486] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.239019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.240017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.241018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.242608] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.244885] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.247043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.251947] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.258044] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.274957] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.279014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.285632] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.294017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.299019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.313026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.324000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.337020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.347016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.373018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.386854] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.396021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.407019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.432016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.448330] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.457018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.468021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.496019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.514520] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.524016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.536018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.566018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.579648] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.589016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.599016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.622017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.632306] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.634492] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.637304] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.639289] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.640124] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.644068] iommu: Default domain type: Passthrough [ 0.645416] SCSI subsystem initialized [ 0.646000] ACPI: bus type USB registered [ 0.648076] usbcore: registered new interface driver usbfs [ 0.649070] usbcore: registered new interface driver hub [ 0.651058] usbcore: registered new device driver usb [ 0.652184] pps_core: LinuxPPS API ver. 1 registered [ 0.654014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.657046] PTP clock support registered [ 0.658261] EDAC MC: Ver: 3.0.0 [ 0.660115] PCI: Using ACPI for IRQ routing [ 0.663577] NetLabel: Initializing [ 0.665011] NetLabel: domain hash size = 128 [ 0.666006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.668068] NetLabel: unlabeled traffic allowed by default [ 0.669461] vgaarb: loaded [ 0.671046] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.672007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.677663] clocksource: Switched to clocksource kvm-clock [ 0.775918] VFS: Disk quotas dquot_6.6.0 [ 0.777874] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.780528] *** VALIDATE ramfs *** [ 0.781844] *** VALIDATE hugetlbfs *** [ 0.783529] pnp: PnP ACPI init [ 0.785910] pnp: PnP ACPI: found 6 devices [ 0.813085] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.815759] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.817425] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.818970] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.821533] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.824055] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.826921] NET: Registered protocol family 2 [ 0.829437] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.834239] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.837891] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.843112] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.846416] TCP: Hash tables configured (established 65536 bind 65536) [ 0.849105] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.852097] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854847] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.858711] NET: Registered protocol family 1 [ 0.861227] RPC: Registered named UNIX socket transport module. [ 0.863046] RPC: Registered udp transport module. [ 0.865329] RPC: Registered tcp transport module. [ 0.867277] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.869856] NET: Registered protocol family 44 [ 0.870975] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.872653] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.874647] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.877976] PCI: CLS 0 bytes, default 64 [ 0.879708] Unpacking initramfs... [ 2.351846] debug: unmapping init [mem 0xffff94067cc54000-0xffff94067ffbffff] [ 2.355657] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.357815] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.360598] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.917868] Initialise system trusted keyrings [ 2.919542] Key type blacklist registered [ 2.921428] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.930745] zbud: loaded [ 2.946726] *** VALIDATE nfs *** [ 2.948757] *** VALIDATE nfs4 *** [ 2.951387] pstore: using deflate compression [ 2.956880] Platform Keyring initialized [ 3.069360] NET: Registered protocol family 38 [ 3.071562] Key type asymmetric registered [ 3.073416] Asymmetric key parser 'x509' registered [ 3.075582] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.078978] io scheduler mq-deadline registered [ 3.080807] io scheduler kyber registered [ 3.083234] io scheduler bfq registered [ 3.085142] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.089441] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.092524] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.095781] ACPI: Power Button [PWRF] [ 3.101852] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.110955] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.146563] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.159274] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.183507] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.213625] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.243969] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.279578] Non-volatile memory driver v1.3 [ 3.282912] Linux agpgart interface v0.103 [ 3.317311] virtio_blk virtio1: [vda] 146944 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.320541] vda: detected capacity change from 0 to 75235328 [ 3.338379] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.341904] vdb: detected capacity change from 0 to 1073741824 [ 3.362893] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.366335] vdc: detected capacity change from 0 to 2621440000 [ 3.384587] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.387945] vdd: detected capacity change from 0 to 2621440000 [ 3.405409] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.408419] vde: detected capacity change from 0 to 4294967296 [ 3.428363] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.430509] vdf: detected capacity change from 0 to 4294967296 [ 3.445783] libphy: Fixed MDIO Bus: probed [ 3.454641] usbcore: registered new interface driver usbserial_generic [ 3.458123] usbserial: USB Serial support registered for generic [ 3.459937] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.463853] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.466074] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.468896] mousedev: PS/2 mouse device common for all mice [ 3.471790] rtc_cmos 00:05: RTC can wake from S4 [ 3.474688] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.478492] rtc_cmos 00:05: registered as rtc0 [ 3.480452] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.481643] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.487546] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.488722] intel_pstate: CPU model not supported [ 3.495904] hid: raw HID events driver (C) Jiri Kosina [ 3.498382] usbcore: registered new interface driver usbhid [ 3.500661] usbhid: USB HID core driver [ 3.502666] drop_monitor: Initializing network drop monitor service [ 3.505370] Initializing XFRM netlink socket [ 3.507657] NET: Registered protocol family 10 [ 3.511339] Segment Routing with IPv6 [ 3.512786] NET: Registered protocol family 17 [ 3.514974] mpls_gso: MPLS GSO support [ 3.520522] RAS: Correctable Errors collector initialized. [ 3.522419] AVX version of gcm_enc/dec engaged. [ 3.523937] AES CTR mode by8 optimization enabled [ 3.605389] sched_clock: Marking stable (3605358740, 0)->(4690742543, -1085383803) [ 3.609514] registered taskstats version 1 [ 3.611904] Loading compiled-in X.509 certificates [ 3.614573] zswap: loaded using pool lzo/zbud [ 3.641552] Key type big_key registered [ 3.656455] Key type encrypted registered [ 3.658297] ima: No TPM chip found, activating TPM-bypass! [ 3.660888] ima: Allocated hash algorithm: sha1 [ 3.662814] ima: No architecture policies found [ 3.664692] evm: Initialising EVM extended attributes: [ 3.666758] evm: security.selinux [ 3.668026] evm: security.ima [ 3.669102] evm: security.capability [ 3.670521] evm: HMAC attrs: 0x1 [ 3.673233] rtc_cmos 00:05: setting system clock to 2026-08-27 14:31:21 UTC (1787841081) [ 3.679971] debug: unmapping init [mem 0xffffffffa6a03000-0xffffffffa6bfffff] [ 3.683603] debug: unmapping init [mem 0xffffffffa5782000-0xffffffffa5a58fff] [ 3.691257] Write protecting the kernel read-only data: 28672k [ 3.693855] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 3.695972] debug: unmapping init [mem 0xffffffffa4714000-0xffffffffa47fffff] [ 3.733251] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.744166] systemd[1]: Detected virtualization kvm. [ 3.746583] systemd[1]: Detected architecture x86-64. [ 3.749102] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.775171] systemd[1]: No hostname configured. [ 3.777771] systemd[1]: Set hostname to . [ 3.781321] random: systemd: uninitialized urandom read (16 bytes read) [ 3.785125] systemd[1]: Initializing machine ID from random generator. [ 3.836701] random: ln: uninitialized urandom read (6 bytes read) [ 4.007331] random: systemd: uninitialized urandom read (16 bytes read) [ 4.011507] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.022973] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.031904] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.932516] device-mapper: uevent: version 1.0.3 [ 4.935587] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.801565] random: fast init done [ 5.810686] virtio_net virtio0 ens2: renamed from eth0 [ 5.847177] scsi host0: ata_piix [ 5.880580] scsi host1: ata_piix [ 5.883753] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.888887] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.182995] random: crng init done [ 10.185026] random: 7 urandom warning(s) missed due to ratelimiting [ 10.934923] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.965980] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.296136] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.571485] SELinux: Disabled at runtime. [ 13.636951] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.647373] systemd[1]: Detected virtualization kvm. [ 13.649145] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.178732] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.182589] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.191106] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.197952] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.203015] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.215435] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.224115] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmat[ 14.558255] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS s File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 15.014959] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.911613] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.938350] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.402480] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.086916] EDAC sbridge: Ver: 1.1.2 [ 19.078706] Key type dns_resolver registered [ 19.455798] NFS: Registering the id_resolver key type [ 19.457874] Key type id_resolver registered [ 19.459622] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg254-server login: [ 52.322720] hrtimer: interrupt took 5241516 ns [ 95.365584] libcfs: loading out-of-tree module taints kernel. [ 95.448765] Key type ._llcrypt registered [ 95.450715] Key type .llcrypt registered [ 95.546985] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 115.444331] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 117.534584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.596624] alg: No test for adler32 (adler32-zlib) [ 119.429626] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 120.231239] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 122.127197] Key type lgssc registered [ 124.267878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 146.100848] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 192.213951] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 207.684576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 207.714084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.141596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.164600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.271994] Lustre: lustre-MDT0000: new disk, initializing [ 209.415860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.449297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 215.079724] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.905801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.023050] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 232.053726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 232.058227] Lustre: Skipped 1 previous similar message [ 232.141862] Lustre: lustre-MDT0001: new disk, initializing [ 232.203349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.235193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 232.254883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 237.132183] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.221497] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 255.715860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.004092] Lustre: lustre-OST0000: new disk, initializing [ 256.008908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.020753] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.163098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 257.340833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 257.372638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 257.481806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 264.712785] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.595054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 283.820253] Lustre: lustre-OST0001: new disk, initializing [ 283.829736] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 283.835889] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 283.924733] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 291.402548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 291.415948] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 291.529164] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.544148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 306.307120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 314.882491] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 325.129857] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing check_logdir /tmp/testlogs/ [ 331.940728] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing yml_node [ 337.077606] Lustre: DEBUG MARKER: Client: 2.17.57.97 [ 340.368281] Lustre: DEBUG MARKER: MDS: 2.17.57.97 [ 342.861345] Lustre: DEBUG MARKER: OSS: 2.17.57.97 [ 344.660403] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:37:00 EDT 2026 [ 367.596469] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 369.340686] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 386.022701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 386.039244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.065887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 389.088568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 389.089407] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.101415] Lustre: Skipped 2 previous similar messages [ 389.130951] Lustre: Skipped 2 previous similar messages [ 394.218196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 394.226556] Lustre: Skipped 1 previous similar message [ 399.327332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 399.328056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 399.352110] Lustre: Skipped 3 previous similar messages [ 399.543773] Lustre: server umount lustre-MDT0000 complete [ 403.734806] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841481 with bad export cookie 4857280873435015098 [ 403.745111] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 403.751184] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 404.451983] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.463753] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 404.474734] Lustre: Skipped 1 previous similar message [ 404.488189] Lustre: Skipped 1 previous similar message [ 413.605663] LustreError: 6529:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 413.611132] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 413.627332] LustreError: 6529:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 413.651404] Lustre: Skipped 3 previous similar messages [ 418.271128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 418.506386] Lustre: server umount lustre-MDT0001 complete [ 422.567615] Lustre: server umount lustre-OST0000 complete [ 426.692856] Lustre: server umount lustre-OST0001 complete [ 434.870545] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 448.510800] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 491.657979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.057862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 492.149431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 492.243384] Lustre: lustre-MDT0000: new disk, initializing [ 492.308468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.330432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 498.796897] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 513.978469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.165134] Lustre: 15099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 514.250215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 514.255747] Lustre: Skipped 1 previous similar message [ 514.393946] Lustre: lustre-MDT0001: new disk, initializing [ 514.542787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 514.600766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 514.619386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 520.083245] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 529.839894] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.724482] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 546.454719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.904953] Lustre: lustre-OST0000: new disk, initializing [ 546.911748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 546.929136] Lustre: 17069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 547.091406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.478905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 548.493960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 548.676707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 555.147391] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 567.997872] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 573.759976] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 574.084800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 579.222387] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 579.528254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 584.680667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.682047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 584.689099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 584.702977] Lustre: Skipped 1 previous similar message [ 587.783751] Lustre: server umount lustre-OST0000 complete [ 591.342846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 591.353864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.840650] Lustre: server umount lustre-MDT0000 complete [ 600.036336] LustreError: 15945:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.051685] LustreError: 15945:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 601.653298] LustreError: 17068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841679 with bad export cookie 4857280873435016631 [ 601.661604] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.662015] LustreError: 17068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 601.927978] Lustre: server umount lustre-MDT0001 complete [ 606.831574] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:41:22 (1787841682) [ 609.602520] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 611.711751] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:41:27 (1787841687) [ 619.258081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.702593] LustreError: 19480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.800864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.501617] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 625.131093] LustreError: 19480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.217949] LustreError: 19481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.446995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.627233] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 648.028819] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 653.536354] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 661.253621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 668.095967] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 676.355204] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 689.633308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 689.640573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.648129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 689.661722] Lustre: Skipped 2 previous similar messages [ 689.682466] Lustre: Skipped 5 previous similar messages [ 693.077810] Lustre: server umount lustre-OST0000 complete [ 712.159902] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 712.487657] Lustre: server umount lustre-MDT0000 complete [ 715.232157] LustreError: 19481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 715.241481] LustreError: 19481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 716.589508] LustreError: 19461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841794 with bad export cookie 4857280873435017513 [ 716.593301] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 716.607578] LustreError: 19461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 717.102733] Lustre: server umount lustre-MDT0001 complete [ 730.210613] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 733.190491] Key type lgssc unregistered [ 733.526528] LNet: 23018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.536996] LNetError: 23018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.553781] LNet: Removed LNI 192.168.202.154@tcp [ 734.561155] Key type .llcrypt unregistered [ 734.564180] Key type ._llcrypt unregistered [ 760.187399] Key type ._llcrypt registered [ 760.190057] Key type .llcrypt registered [ 760.354899] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 762.226772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 762.565749] alg: No test for adler32 (adler32-zlib) [ 763.659716] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 764.091409] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 765.890229] Key type lgssc registered [ 767.216315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.236992] loop: module loaded [ 777.350996] loop0: detected capacity change from 0 to 204800000 [ 777.713362] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 777.728041] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 778.209852] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 785.532909] loop0: detected capacity change from 0 to 204800000 [ 785.751744] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 785.762708] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 785.774056] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 792.417854] loop0: detected capacity change from 0 to 204800000 [ 799.352727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 799.387684] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.811436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 800.926173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 801.027208] Lustre: lustre-MDT0000: new disk, initializing [ 801.163443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 801.205818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 806.832327] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 817.528618] loop1: detected capacity change from 0 to 204800000 [ 823.041556] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 823.614630] Lustre: lustre-OST07c6: new disk, initializing [ 823.626604] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 823.633240] Lustre: Skipped 1 previous similar message [ 823.642191] Lustre: 26206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 823.788138] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 833.762695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 833.774875] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 833.834741] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 834.141979] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 842.438549] Lustre: 26709:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 842.450316] Lustre: 26709:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 842.460488] Lustre: 26709:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 842.481155] Lustre: 26709:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 842.490182] Lustre: 26709:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 842.501682] Lustre: 26709:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 866.911870] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 866.920874] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 866.935205] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 866.947242] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 866.954653] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 866.962934] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 872.511597] Lustre: 25195:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 872.520513] Lustre: 25195:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 872.526228] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 872.531245] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.537561] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 872.554231] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.565732] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 872.570687] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.576540] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 872.583011] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.590835] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 872.605941] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 875.922457] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 875.932724] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 875.939167] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 875.950704] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 875.960930] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 875.974947] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 875.980416] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 875.986447] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 875.990938] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 875.995390] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 876.000726] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 876.005469] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 881.266917] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 881.279608] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 881.289459] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 881.298043] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 881.307271] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 881.314768] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 881.324458] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 881.331802] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 881.340449] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 881.350870] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 881.355226] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 881.364361] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 890.232190] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 890.247189] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 890.269070] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 890.279085] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 890.284945] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 890.291666] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 890.297567] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 890.301509] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 890.305360] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 890.308290] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 890.315568] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 890.325571] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 946.656191] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 946.673532] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.693083] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 947.889199] Lustre: server umount lustre-OST07c6 complete [ 958.238039] Lustre: server umount lustre-MDT0000 complete [ 978.378831] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 981.833458] Key type lgssc unregistered [ 982.122293] LNet: 28770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.130519] LNetError: 28770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.228564] LNet: Removed LNI 192.168.202.154@tcp [ 984.312473] Key type .llcrypt unregistered [ 984.316418] Key type ._llcrypt unregistered [ 1007.275180] Key type ._llcrypt registered [ 1007.277276] Key type .llcrypt registered [ 1007.398793] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1008.795847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.822847] alg: No test for adler32 (adler32-zlib) [ 1010.087550] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 1010.552531] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1012.295273] Key type lgssc registered [ 1013.695906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1022.703559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1022.721827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.583449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.895062] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1034.729156] LustreError: 30044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.768424] LustreError: 30044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1039.638194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.841360] LustreError: 30045:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.082236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1045.410498] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1054.570206] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1061.841294] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1069.226858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1069.608558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.581257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1076.800597] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1085.122356] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1101.285821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1101.306397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.338141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1103.256596] Lustre: server umount lustre-OST0000 complete [ 1110.497586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1110.531757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1110.541498] Lustre: Skipped 1 previous similar message [ 1113.455089] Lustre: server umount lustre-MDT0000 complete [ 1115.625438] LustreError: 30044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.654027] LustreError: 30044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1117.563152] LustreError: 32829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842195 with bad export cookie 15583460681193749046 [ 1117.581944] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1128.148400] Lustre: server umount lustre-MDT0001 complete [ 1142.077527] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1145.090839] Key type lgssc unregistered [ 1145.360598] LNet: 33786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.365619] LNetError: 33786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.408863] LNet: Removed LNI 192.168.202.154@tcp [ 1146.616945] Key type .llcrypt unregistered [ 1146.621426] Key type ._llcrypt unregistered [ 1167.641775] Key type ._llcrypt registered [ 1167.644265] Key type .llcrypt registered [ 1167.860462] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1169.770621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1169.892779] alg: No test for adler32 (adler32-zlib) [ 1171.082455] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 1171.375687] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1173.063234] Key type lgssc registered [ 1174.378368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1183.547051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1183.594385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.128896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.590846] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1194.477721] LustreError: 35056:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1194.492934] LustreError: 35056:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1199.589579] LustreError: 35057:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.895276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.255563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1206.360200] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1214.334937] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1220.585178] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1227.639832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1228.014995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.095123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1235.372215] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1244.943726] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1255.907961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.918590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1255.924171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1255.939728] LustreError: Skipped 1 previous similar message [ 1261.036940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.042919] Lustre: Skipped 2 previous similar messages [ 1262.084938] Lustre: server umount lustre-OST0000 complete [ 1266.153761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.160884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.166483] Lustre: Skipped 2 previous similar messages [ 1266.184750] Lustre: Skipped 1 previous similar message [ 1271.276966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.295873] Lustre: Skipped 1 previous similar message [ 1272.372412] Lustre: server umount lustre-MDT0000 complete [ 1276.395482] LustreError: 35053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.422616] LustreError: 35053:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1276.818109] LustreError: 35035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842354 with bad export cookie 2768375439480799163 [ 1276.837448] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1276.842730] LustreError: 35035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1277.430607] Lustre: server umount lustre-MDT0001 complete [ 1290.267066] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1293.296556] Key type lgssc unregistered [ 1293.590575] LNet: 38597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.598527] LNetError: 38597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.613511] LNet: Removed LNI 192.168.202.154@tcp [ 1294.421352] Key type .llcrypt unregistered [ 1294.423780] Key type ._llcrypt unregistered [ 1319.331803] Key type ._llcrypt registered [ 1319.333946] Key type .llcrypt registered [ 1319.451076] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1320.781196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.938663] alg: No test for adler32 (adler32-zlib) [ 1321.965710] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 1322.286900] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1324.056129] Key type lgssc registered [ 1325.581203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1333.853855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1333.867327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.391594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.679156] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1345.514308] LustreError: 40016:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.520894] LustreError: 40016:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1349.365583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.774860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1354.549802] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1363.125523] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1369.752294] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1377.389735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1377.778903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.855267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1384.917619] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1393.415503] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1400.290712] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.294425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1400.325479] Lustre: Skipped 2 previous similar messages [ 1400.346855] Lustre: Skipped 2 previous similar messages [ 1405.419669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1405.432707] Lustre: Skipped 3 previous similar messages [ 1410.529219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.546056] Lustre: Skipped 3 previous similar messages [ 1414.114132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1414.344390] Lustre: server umount lustre-MDT0000 complete [ 1415.654333] LustreError: 40016:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.678244] LustreError: 40016:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1418.547967] LustreError: 39997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842496 with bad export cookie 1083976494442038942 [ 1418.561670] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1418.564472] LustreError: 39997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1420.776186] LustreError: 40011:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.789946] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.811115] LustreError: 40011:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1420.816400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1420.816408] Lustre: Skipped 1 previous similar message [ 1423.776196] LustreError: 40012:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.785641] LustreError: 40012:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1424.966565] Lustre: server umount lustre-MDT0001 complete [ 1440.980866] Lustre: server umount lustre-OST0000 complete [ 1462.089979] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1466.202985] Key type lgssc unregistered [ 1466.538853] LNet: 43879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.547667] LNetError: 43879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.561387] LNet: Removed LNI 192.168.202.154@tcp [ 1467.760459] Key type .llcrypt unregistered [ 1467.762566] Key type ._llcrypt unregistered [ 1495.195943] Key type ._llcrypt registered [ 1495.197835] Key type .llcrypt registered [ 1495.271730] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1496.043605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1496.090388] alg: No test for adler32 (adler32-zlib) [ 1497.212723] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 1497.593844] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1499.247337] Key type lgssc registered [ 1500.439635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1511.216228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1511.243528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.837970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.187304] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1523.174943] LustreError: 45297:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1523.229722] LustreError: 45297:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1527.189702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.565140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1533.199719] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1540.902895] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1545.902819] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1553.722655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1554.149966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1558.259187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1562.131211] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1568.973108] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1577.957328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.961618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1577.966382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1577.971475] Lustre: Skipped 1 previous similar message [ 1577.987853] Lustre: Skipped 1 previous similar message [ 1579.716841] Lustre: server umount lustre-OST0000 complete [ 1599.456434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1599.465940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.479254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1602.532618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1602.535094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.568838] Lustre: server umount lustre-MDT0000 complete [ 1607.648910] LustreError: 45296:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.670512] LustreError: 45296:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1608.450986] LustreError: 45278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842686 with bad export cookie 9185438623072799220 [ 1608.456306] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1608.474045] LustreError: 45278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1614.910985] Lustre: server umount lustre-MDT0001 complete [ 1626.107833] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1628.538286] Key type lgssc unregistered [ 1628.849323] LNet: 49007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.857706] LNetError: 49007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.870583] LNet: Removed LNI 192.168.202.154@tcp [ 1629.743168] Key type .llcrypt unregistered [ 1629.747328] Key type ._llcrypt unregistered [ 1655.271096] Key type ._llcrypt registered [ 1655.277333] Key type .llcrypt registered [ 1655.443409] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1656.705187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1656.958957] alg: No test for adler32 (adler32-zlib) [ 1658.211170] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 1658.597692] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1660.336419] Key type lgssc registered [ 1661.985751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1670.326592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1670.349759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.795930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.054376] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1681.380525] LustreError: 50424:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.397781] LustreError: 50424:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1686.499044] LustreError: 50425:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.414165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.989763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1692.489748] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1700.987402] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1707.395171] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1715.044740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1715.486875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.539055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1723.440530] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1732.200285] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1739.867382] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1744.444907] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1757.558217] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2711h00m32s [ 1759.534965] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1763.814367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.818047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1763.837169] Lustre: Skipped 2 previous similar messages [ 1763.864936] Lustre: Skipped 2 previous similar messages [ 1764.898793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1764.905125] Lustre: Skipped 2 previous similar messages [ 1766.715131] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1766.731675] Lustre: Skipped 1 previous similar message [ 1767.871371] Lustre: server umount lustre-MDT0000 complete [ 1770.985747] LustreError: 50425:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.006428] LustreError: 50425:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1772.243363] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1777.127917] LustreError: 50419:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1777.154058] LustreError: 50419:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1779.070709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.272252] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1779.524614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.603420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1780.529482] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1785.861331] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1790.422894] Lustre: DEBUG MARKER: Wait for df (36726) ... [ 1792.629871] Lustre: DEBUG MARKER: done [ 1799.652717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1799.672440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.692839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.165467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1805.176501] Lustre: server umount lustre-OST0000 complete [ 1811.430456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1811.434267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.443097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1811.443108] Lustre: Skipped 3 previous similar messages [ 1811.469407] Lustre: Skipped 2 previous similar messages [ 1824.223207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1824.436604] Lustre: server umount lustre-MDT0000 complete [ 1824.738763] LustreError: 50425:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1824.746288] LustreError: 50425:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1828.496862] LustreError: 51159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842906 with bad export cookie 16001956850507991511 [ 1828.502090] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1828.513681] LustreError: 51159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1828.869523] Lustre: server umount lustre-MDT0001 complete [ 1841.078234] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1845.074473] Key type lgssc unregistered [ 1845.452233] LNet: 55370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.460375] LNetError: 55370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.473691] LNet: Removed LNI 192.168.202.154@tcp [ 1846.424660] Key type .llcrypt unregistered [ 1846.426598] Key type ._llcrypt unregistered [ 1890.454466] Key type ._llcrypt registered [ 1890.456788] Key type .llcrypt registered [ 1890.527296] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1892.065228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1892.122316] alg: No test for adler32 (adler32-zlib) [ 1893.242968] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 1893.452234] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1895.112884] Key type lgssc registered [ 1896.313120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1904.455419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1904.471666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.873547] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1905.892322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1906.138709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.591466] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1922.059389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.208495] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1922.245935] Lustre: 57577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1922.630692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1927.600176] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1937.801708] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1944.740264] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1952.520347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1952.774486] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1953.012261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.792971] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 1962.133335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1969.486437] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1977.296439] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2711h04m13s [ 1978.919922] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1987.307786] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1990.504428] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1990.544397] LustreError: 59292:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2010.947856] Lustre: lustre-MDT0000: Client 8fe554e8-688b-4aec-814a-6b55b6e48caf (at 192.168.202.54@tcp) reconnecting [ 2030.615145] LustreError: 59292:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2032.958197] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2035.913991] Lustre: DEBUG MARKER: done [ 2043.872386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.878086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.896190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.385377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2044.388639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.419527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.423177] Lustre: Skipped 1 previous similar message [ 2048.422596] Lustre: server umount lustre-OST0000 complete [ 2054.624926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2054.635670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2054.640671] Lustre: Skipped 1 previous similar message [ 2054.650314] Lustre: Skipped 1 previous similar message [ 2059.137454] Lustre: server umount lustre-MDT0000 complete [ 2063.119315] LustreError: 59440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843140 with bad export cookie 16676531577019854783 [ 2063.121945] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2063.130301] LustreError: 59440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2063.636842] Lustre: server umount lustre-MDT0001 complete [ 2074.317440] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2076.671647] Key type lgssc unregistered [ 2076.925937] LNet: 61912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.933478] LNetError: 61912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.995689] LNet: Removed LNI 192.168.202.154@tcp [ 2078.605882] Key type .llcrypt unregistered [ 2078.608939] Key type ._llcrypt unregistered [ 2128.141953] Key type ._llcrypt registered [ 2128.144800] Key type .llcrypt registered [ 2128.262440] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2129.583646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.727881] alg: No test for adler32 (adler32-zlib) [ 2130.728517] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 2130.946279] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2132.615911] Key type lgssc registered [ 2133.824660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2137.853842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2137.888339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.411599] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2139.449712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2139.747391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.002650] Lustre: Failing over lustre-MDT0000 [ 2141.433935] Lustre: server umount lustre-MDT0000 complete [ 2149.253353] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:07:05 (1787843225) [ 2156.565794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.052627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.424187] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2171.272321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.336198] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2171.370392] Lustre: 64714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2171.638150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2176.305233] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2184.354778] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2190.808537] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2197.716854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2197.882195] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2198.055529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.195299] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2207.851475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2213.304421] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2219.236528] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2228.199524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2228.221686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.261323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.847522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.856881] Lustre: Skipped 4 previous similar messages [ 2234.884813] Lustre: server umount lustre-MDT0000 complete [ 2240.404284] LustreError: 64704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843318 with bad export cookie 10373032833885527098 [ 2240.429969] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.841243] Lustre: server umount lustre-MDT0001 complete [ 2245.273510] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2249.971471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2258.751149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.423179] Lustre: 63633:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843321/real 1787843321] req@000000003fea70d1 x1874689479713536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843337 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2259.461360] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.474715] Lustre: Skipped 2 previous similar messages [ 2260.447197] Lustre: 63634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843322/real 1787843322] req@000000006e232ec5 x1874689479713664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2264.543197] Lustre: 63636:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843326/real 1787843326] req@00000000030fc3a4 x1874689479713792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2265.646576] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8ff46a675b35e51d [ 2265.682455] Lustre: MGC192.168.202.154@tcp: Connection restored to 0@lo (at 0@lo) [ 2266.084363] LustreError: 66561:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.096674] LustreError: 66561:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2266.155342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.256053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2267.171836] Lustre: 63635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843328/real 1787843328] req@000000009774fb37 x1874689479714048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2268.194594] LustreError: 66562:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.208317] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2269.293640] LustreError: 68483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.319109] LustreError: 68483:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2271.345144] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2272.247989] Lustre: 63634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843333/real 1787843333] req@00000000b085d711 x1874689479714176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2274.788590] LustreError: 68481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.826491] LustreError: 68481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2279.907093] LustreError: 68483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.936185] LustreError: 68483:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2280.400720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.827311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.043555] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2286.062803] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2293.242765] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2299.470352] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2311.659239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.661909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.666263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2311.698039] Lustre: Skipped 2 previous similar messages [ 2311.724820] Lustre: Skipped 2 previous similar messages [ 2313.487462] Lustre: server umount lustre-MDT0000 complete [ 2316.779375] LustreError: 66561:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.803375] LustreError: 66561:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2318.221587] LustreError: 64706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843396 with bad export cookie 10373032833885529373 [ 2318.223974] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2318.235156] LustreError: 64706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2318.534636] Lustre: server umount lustre-MDT0001 complete [ 2324.143996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2334.650896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.247191] Lustre: 63633:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843399/real 1787843399] req@000000009aecf9d0 x1874689479742208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2337.288775] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.823511] LustreError: 69218:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.893065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.948820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2349.127523] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2357.984246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.370540] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2363.539235] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2372.909073] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2379.138417] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2385.980877] Lustre: server umount lustre-MDT0000 complete [ 2386.919770] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.934306] LustreError: 69218:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.962077] LustreError: 69218:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 2389.902386] LustreError: 67624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843467 with bad export cookie 10373032833885530752 [ 2389.905581] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.919448] LustreError: 67624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2390.349622] Lustre: server umount lustre-MDT0001 complete [ 2396.858894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2400.333662] Lustre: DEBUG MARKER: files compared the same [ 2412.876455] Lustre: server umount lustre-OST0000 complete [ 2432.393629] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2435.366674] Key type lgssc unregistered [ 2435.681649] LNet: 74445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.692843] LNetError: 74445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.714117] LNet: Removed LNI 192.168.202.154@tcp [ 2436.555269] Key type .llcrypt unregistered [ 2436.557420] Key type ._llcrypt unregistered [ 2464.240160] Key type ._llcrypt registered [ 2464.244781] Key type .llcrypt registered [ 2464.383844] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2465.757835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2465.996566] alg: No test for adler32 (adler32-zlib) [ 2467.091977] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 2467.410434] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2469.159487] Key type lgssc registered [ 2470.277965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2477.494487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2477.530154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.067572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.856919] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug +malloc all [ 2488.289919] LustreError: 75865:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.304248] LustreError: 75865:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2492.701587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.041307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2497.305953] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug +malloc all [ 2504.704339] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2510.271580] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2517.403448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2517.668986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2523.844553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2524.730085] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug +malloc all [ 2532.532751] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2544.608157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.614346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2544.618937] Lustre: Skipped 1 previous similar message [ 2544.633085] Lustre: Skipped 1 previous similar message [ 2547.786891] Lustre: server umount lustre-OST0000 complete [ 2554.340236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.343449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.374305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.577843] Lustre: server umount lustre-MDT0000 complete [ 2559.968085] LustreError: 75865:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.992827] LustreError: 75865:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2562.362372] LustreError: 75846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843640 with bad export cookie 12279861541433170248 [ 2562.368509] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.378535] LustreError: 75846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2562.980482] Lustre: server umount lustre-MDT0001 complete [ 2574.903792] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2577.783926] Key type lgssc unregistered [ 2578.061769] LNet: 79409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.072990] LNetError: 79409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2578.081651] LNet: Removed LNI 192.168.202.154@tcp [ 2578.933403] Key type .llcrypt unregistered [ 2578.935609] Key type ._llcrypt unregistered [ 2605.113257] Key type ._llcrypt registered [ 2605.115343] Key type .llcrypt registered [ 2605.241183] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2606.818859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2606.844074] alg: No test for adler32 (adler32-zlib) [ 2607.913836] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 2608.154088] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2609.871801] Key type lgssc registered [ 2611.297625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2619.109264] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2619.133947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2634.719928] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2639.839450] LustreError: 80802:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 2666.591221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.951989] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2683.790710] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2691.735756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.879394] Lustre: *** cfs_fail_loc=706, val=0*** [ 2692.221502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.293955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2697.236615] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2697.697837] LustreError: 80827:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.707532] LustreError: 80827:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2701.838023] LustreError: 81886:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.871114] LustreError: 81886:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2706.813867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.913846] LustreError: 81887:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.934301] LustreError: 81887:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2707.216928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2712.433571] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2721.274554] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2727.389081] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2733.028548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.032569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2733.050839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.067491] LustreError: Skipped 1 previous similar message [ 2733.088565] Lustre: Skipped 1 previous similar message [ 2735.810314] Lustre: server umount lustre-OST0000 complete [ 2743.265917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.269352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2743.279662] Lustre: Skipped 2 previous similar messages [ 2743.288244] Lustre: Skipped 1 previous similar message [ 2745.859873] Lustre: server umount lustre-MDT0000 complete [ 2748.390532] LustreError: 81891:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.413622] LustreError: 81891:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2750.654630] LustreError: 80810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843828 with bad export cookie 10240648285372541328 [ 2750.659412] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.663184] LustreError: 80810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2751.159652] Lustre: server umount lustre-MDT0001 complete [ 2763.727237] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2767.101734] Key type lgssc unregistered [ 2767.599073] LNet: 84420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2767.619577] LNetError: 84420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2767.668344] LNet: Removed LNI 192.168.202.154@tcp [ 2768.729684] Key type .llcrypt unregistered [ 2768.731558] Key type ._llcrypt unregistered [ 2801.266488] Key type ._llcrypt registered [ 2801.270258] Key type .llcrypt registered [ 2801.393415] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2802.990875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2803.049420] alg: No test for adler32 (adler32-zlib) [ 2804.454252] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 2804.696042] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2806.415224] Key type lgssc registered [ 2808.426875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2817.019204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2817.033851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.806396] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2830.967386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2831.681095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2836.460812] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2836.963640] LustreError: 86529:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.982624] LustreError: 86529:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2842.085182] LustreError: 86528:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.098637] LustreError: 86528:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2845.589027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2845.888160] LustreError: 86529:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.903791] LustreError: 86529:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2845.909390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.303276] LustreError: 86529:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.324725] LustreError: 86529:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2852.505857] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2856.423975] LustreError: 87277:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.445864] LustreError: 87277:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2863.210619] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2863.223213] Lustre: MGS: Client 07abbccb-7aaf-4fbf-a3c3-eacb80aedb80 (at 0@lo) reconnecting [ 2863.249241] LustreError: 86543:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2863.421587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2863.481258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2867.898976] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 2884.075680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.089222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2884.089717] Lustre: Skipped 1 previous similar message [ 2884.098396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2884.104020] Lustre: Skipped 1 previous similar message [ 2889.188253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2889.199283] Lustre: Skipped 1 previous similar message [ 2890.164867] Lustre: server umount lustre-OST0000 complete [ 2899.425940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.436478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2899.445954] Lustre: Skipped 1 previous similar message [ 2901.018558] Lustre: server umount lustre-MDT0000 complete [ 2904.545398] LustreError: 86533:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.572023] LustreError: 86533:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2905.801331] Lustre: server umount lustre-MDT0001 complete [ 2910.110978] Lustre: server umount MGS complete [ 2923.365621] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:19:59 (1787843999) [ 2957.607882] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 2968.068361] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3005.503484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.776464] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3019.944065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.227721] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3020.297619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3020.393305] Lustre: lustre-MDT0001: new disk, initializing [ 3020.464611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3020.484800] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3020.491649] Lustre: 93721:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3020.506351] LustreError: 93721:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3025.296131] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3036.771154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3037.147864] Lustre: lustre-OST0000: new disk, initializing [ 3037.151374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3037.160186] Lustre: 94486:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3037.219207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.337671] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3045.998675] Lustre: 94862:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004735129b x1874690186454016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787844139 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3054.964318] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3062.168323] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.184354] Lustre: MGS: Client 11e1f061-c99a-40db-88eb-c98cd8582dc1 (at 0@lo) reconnecting [ 3062.256478] Lustre: 95532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3062.274042] LustreError: 93707:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3062.327560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3062.425819] Lustre: lustre-MDT0000: new disk, initializing [ 3062.655519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.679538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3062.709502] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3063.780565] Lustre: 95548:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003fea70d1 x1874690186461056/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787844157 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3063.819602] Lustre: 95548:0:(client.c:1652:after_reply()) Skipped 1 previous similar message [ 3067.459468] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3068.960565] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3072.944694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3072.954985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3073.050279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3101.162701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.171247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3101.199023] Lustre: Skipped 1 previous similar message [ 3101.224935] Lustre: Skipped 2 previous similar messages [ 3106.323792] Lustre: server umount lustre-OST0000 complete [ 3116.512788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3116.519922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.535273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3126.239212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3126.504827] Lustre: server umount lustre-MDT0000 complete [ 3127.264763] LustreError: 93697:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.284138] LustreError: 93697:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3137.067469] Lustre: server umount lustre-MDT0001 complete [ 3141.331152] Lustre: server umount MGS complete [ 3154.222339] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:23:50 (1787844230) [ 3173.793402] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3177.428377] Key type lgssc unregistered [ 3177.701879] LNet: 98061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.721373] LNetError: 98061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3177.738861] LNet: Removed LNI 192.168.202.154@tcp [ 3178.999766] Key type .llcrypt unregistered [ 3179.000852] Key type ._llcrypt unregistered [ 3199.178322] Key type ._llcrypt registered [ 3199.180226] Key type .llcrypt registered [ 3199.346480] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3201.296772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3201.359188] alg: No test for adler32 (adler32-zlib) [ 3202.647284] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 3203.076964] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3204.823188] Key type lgssc registered [ 3206.362726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3215.896770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3215.940727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.502642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.135824] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3227.634094] LustreError: 99283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.661406] LustreError: 99283:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3232.752075] LustreError: 99284:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.492971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.139924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3241.931440] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3250.571589] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3258.721442] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3268.396659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3268.673174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.522966] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3288.139040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3290.662475] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3301.347171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3301.367674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.385387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3303.395261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3303.405512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3303.415397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.435557] Lustre: Skipped 1 previous similar message [ 3307.233610] Lustre: server umount lustre-OST0000 complete [ 3313.139299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3313.150160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.169783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3318.617946] Lustre: server umount lustre-MDT0000 complete [ 3318.766837] LustreError: 99283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.787857] LustreError: 99283:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3323.055061] LustreError: 99265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844400 with bad export cookie 11955491936129737244 [ 3323.066542] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3323.067541] LustreError: 99265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3323.648120] Lustre: server umount lustre-MDT0001 complete [ 3336.566871] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3339.712799] Key type lgssc unregistered [ 3340.110233] LNet: 102823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3340.128675] LNetError: 102823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3340.157514] LNet: Removed LNI 192.168.202.154@tcp [ 3341.349224] Key type .llcrypt unregistered [ 3341.354948] Key type ._llcrypt unregistered [ 3361.042780] Key type ._llcrypt registered [ 3361.044492] Key type .llcrypt registered [ 3361.171421] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3363.277160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3363.314655] alg: No test for adler32 (adler32-zlib) [ 3364.485772] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 3364.785135] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3366.543400] Key type lgssc registered [ 3368.040713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3375.524754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3375.537893] LustreError: 103911:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3375.539403] Lustre: Skipped 1 previous similar message [ 3375.539434] LustreError: 103910:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3375.553608] LustreError: 103911:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3375.562617] LustreError: 103911:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3375.563838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.571826] LustreError: 103911:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3377.196358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3386.342101] LustreError: 103949:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.357024] LustreError: 103949:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3386.983743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.451988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3393.010099] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3399.823659] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing lsmod [ 3403.129792] LustreError: 105113:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3403.184787] LustreError: 105120:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3403.190869] LustreError: 105113:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4949 [ 3403.196745] LustreError: 105113:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3403.213681] LustreError: 105113:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3403.218472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3403.555085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.840495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3407.851843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3407.875520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.871097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.880936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.883635] Lustre: Skipped 1 previous similar message [ 3408.902417] Lustre: Skipped 2 previous similar messages [ 3413.026945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.044712] Lustre: Skipped 2 previous similar messages [ 3413.716736] Lustre: server umount lustre-MDT0000 complete [ 3417.912837] LustreError: 103927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844495 with bad export cookie 12739154716081600771 [ 3417.925797] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3417.942831] Lustre: 103941:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3418.081384] LustreError: 103943:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.083396] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.094407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3419.296530] LustreError: 104380:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006e6c8be0 x1874690773634688/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3419.503949] Lustre: server umount lustre-MDT0001 complete [ 3423.902131] Lustre: server umount lustre-OST0000 complete [ 3433.713472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3434.173961] LustreError: 106183:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3434.242302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3439.581570] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3439.589199] LustreError: 106184:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.707640] LustreError: 106183:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.523289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.866436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.521219] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3463.291634] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3469.188427] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3476.787708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3477.241966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.291695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3484.645343] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3493.081662] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3500.007127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3500.017398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.036594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.107362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.122845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.217993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.228896] Lustre: Skipped 3 previous similar messages [ 3514.335196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3514.490304] Lustre: server umount lustre-OST0000 complete [ 3519.458454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.147943] Lustre: server umount lustre-MDT0000 complete [ 3529.459178] LustreError: 106164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844607 with bad export cookie 12739154716081601457 [ 3529.466301] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3529.480239] LustreError: 106164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3529.831248] Lustre: server umount lustre-MDT0001 complete [ 3543.710789] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3546.614679] Key type lgssc unregistered [ 3546.916763] LNet: 109719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3546.921306] LNetError: 109719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3546.933691] LNet: Removed LNI 192.168.202.154@tcp [ 3547.950781] Key type .llcrypt unregistered [ 3547.954203] Key type ._llcrypt unregistered [ 3572.506422] Key type ._llcrypt registered [ 3572.509295] Key type .llcrypt registered [ 3572.713077] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3574.106553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3574.303171] alg: No test for adler32 (adler32-zlib) [ 3575.526345] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 3575.775407] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3577.455646] Key type lgssc registered [ 3578.811811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3587.372600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3587.399058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3588.885924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.981908] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3598.308493] LustreError: 111139:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3598.324477] LustreError: 111139:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3602.772369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3603.186612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3608.697214] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3617.343378] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3624.066801] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3632.601022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3633.117047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.197328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3640.962885] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3650.118612] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3656.945365] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3661.225989] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3667.938040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3667.944259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.955128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.060477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.066841] Lustre: Skipped 2 previous similar messages [ 3678.177802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3678.188702] Lustre: Skipped 1 previous similar message [ 3680.224822] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3680.491250] Lustre: server umount lustre-OST0000 complete [ 3685.345884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3685.364053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.376438] Lustre: Skipped 1 previous similar message [ 3685.385839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3688.421985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.729761] Lustre: server umount lustre-MDT0000 complete [ 3693.538606] LustreError: 112112:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3693.579511] LustreError: 112112:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3695.189992] LustreError: 111119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844773 with bad export cookie 12845023744264031317 [ 3695.194333] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3695.203349] LustreError: 111119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3695.615996] Lustre: server umount lustre-MDT0001 complete [ 3707.516486] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3710.377530] Key type lgssc unregistered [ 3710.652679] LNet: 114783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3710.661055] LNetError: 114783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3710.687246] LNet: Removed LNI 192.168.202.154@tcp [ 3711.500568] Key type .llcrypt unregistered [ 3711.502563] Key type ._llcrypt unregistered [ 3731.780093] Key type ._llcrypt registered [ 3731.781685] Key type .llcrypt registered [ 3731.877433] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3733.429391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3733.549291] alg: No test for adler32 (adler32-zlib) [ 3734.578977] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 3734.829576] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3736.545719] Key type lgssc registered [ 3737.778695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3746.858529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3746.874107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.473818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.305244] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3759.076460] LustreError: 116006:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.090486] LustreError: 116006:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3762.409997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.029955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3768.090808] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3776.168193] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3782.673960] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3790.787837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3791.342195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3793.429786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3799.351605] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3808.621047] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3823.086179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.099158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.101926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3825.730222] Lustre: server umount lustre-OST0000 complete [ 3829.744690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3829.751687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.774962] Lustre: Skipped 1 previous similar message [ 3829.781433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3829.783214] Lustre: Skipped 1 previous similar message [ 3833.313562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.317769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.964331] Lustre: server umount lustre-MDT0000 complete [ 3838.433851] LustreError: 117160:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3838.462166] LustreError: 117160:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3840.366083] LustreError: 115987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844918 with bad export cookie 4456453648545959721 [ 3840.377432] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3840.387749] LustreError: 115987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3840.729848] Lustre: server umount lustre-MDT0001 complete [ 3853.267681] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3856.967810] Key type lgssc unregistered [ 3857.378659] LNet: 119542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3857.385143] LNetError: 119542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3857.406237] LNet: Removed LNI 192.168.202.154@tcp [ 3858.532121] Key type .llcrypt unregistered [ 3858.533716] Key type ._llcrypt unregistered [ 3889.971118] Key type ._llcrypt registered [ 3889.972652] Key type .llcrypt registered [ 3890.055241] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3891.662356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3891.726796] alg: No test for adler32 (adler32-zlib) [ 3892.816681] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 3892.967591] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3894.599139] Key type lgssc registered [ 3895.972940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3905.146226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3905.163808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.773038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.718176] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3916.268670] LustreError: 120962:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3916.308020] LustreError: 120962:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3921.376514] LustreError: 120963:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.586983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.897872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3926.868466] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3935.592217] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3942.981344] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3951.386765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3951.767063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.239280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3959.139980] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 3968.346787] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3976.158500] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3980.498597] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3980.501021] Lustre: Skipped 2 previous similar messages [ 3982.466732] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3982.521191] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3985.743259] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3985.753826] Lustre: Skipped 2 previous similar messages [ 3989.148108] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3989.160996] Lustre: Skipped 1 previous similar message [ 3996.098383] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.54@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3996.117287] Lustre: Skipped 2 previous similar messages [ 3997.171830] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.54@tcp 192.168.0.[2,10]@tcp [ 4001.610396] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.54/24@tcp in log lustre-MDT0000 [ 4001.617024] Lustre: Skipped 2 previous similar messages [ 4003.415174] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.54/24@tcp [ 4003.425469] Lustre: Skipped 1 previous similar message [ 4013.037180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4013.039363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.046389] LustreError: Skipped 1 previous similar message [ 4013.065830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4015.036382] Lustre: server umount lustre-OST0000 complete [ 4022.752921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.755678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4022.766118] Lustre: Skipped 1 previous similar message [ 4022.780264] Lustre: Skipped 1 previous similar message [ 4025.796013] Lustre: server umount lustre-MDT0000 complete [ 4027.879279] LustreError: 120958:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.909488] LustreError: 120958:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4030.525378] LustreError: 120943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845108 with bad export cookie 9612156337503009248 [ 4030.538361] LustreError: 120943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4030.547291] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4041.646187] Lustre: server umount lustre-MDT0001 complete [ 4055.185278] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4058.108907] Key type lgssc unregistered [ 4058.582321] LNet: 125675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4058.590912] LNetError: 125675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4058.619068] LNet: Removed LNI 192.168.202.154@tcp [ 4059.535885] Key type .llcrypt unregistered [ 4059.540450] Key type ._llcrypt unregistered [ 4096.816916] Key type ._llcrypt registered [ 4096.821985] Key type .llcrypt registered [ 4096.930264] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4098.315668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4098.357732] alg: No test for adler32 (adler32-zlib) [ 4099.524372] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 4099.863053] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4101.561387] Key type lgssc registered [ 4102.882244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4111.796300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4111.818198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.356962] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4113.393497] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.54/24@tcp [ 4113.402348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.728510] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4123.620287] LustreError: 127245:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.647167] LustreError: 127245:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4127.638542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.904939] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4127.957236] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.54/24@tcp [ 4128.061858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4133.985924] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4143.993512] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4150.656570] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4158.333783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4158.846511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4159.924953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4166.643075] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4175.087333] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4185.070099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4185.075557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.093256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4187.618084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4187.618365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4187.639801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.734981] Lustre: server umount lustre-OST0000 complete [ 4197.860725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.862251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.880545] Lustre: Skipped 1 previous similar message [ 4197.895128] Lustre: Skipped 2 previous similar messages [ 4201.462384] Lustre: server umount lustre-MDT0000 complete [ 4202.980697] LustreError: 127242:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.011826] LustreError: 127242:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4205.740288] LustreError: 127227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845283 with bad export cookie 5962040413912253782 [ 4205.747734] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4205.757415] LustreError: 127227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4206.129559] Lustre: server umount lustre-MDT0001 complete [ 4219.504898] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4222.994638] Key type lgssc unregistered [ 4223.354796] LNet: 130836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4223.362097] LNetError: 130836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4223.401597] LNet: Removed LNI 192.168.202.154@tcp [ 4224.562417] Key type .llcrypt unregistered [ 4224.570240] Key type ._llcrypt unregistered [ 4254.968327] Key type ._llcrypt registered [ 4254.971265] Key type .llcrypt registered [ 4255.073524] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 4271.210853] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4272.610486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4272.683475] alg: No test for adler32 (adler32-zlib) [ 4273.811979] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 4274.088045] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4275.735293] Key type lgssc registered [ 4277.004984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4313.183339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4313.207130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.616741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4314.677024] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4314.801623] Lustre: lustre-MDT0000: new disk, initializing [ 4314.923264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.943846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4319.963802] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4330.804322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.929180] Lustre: 134238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4330.979171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4330.982841] Lustre: Skipped 1 previous similar message [ 4331.040731] Lustre: lustre-MDT0001: new disk, initializing [ 4331.163815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4331.201641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4331.213986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4336.374285] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4346.909798] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4353.161490] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4360.003678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4360.354747] Lustre: lustre-OST0000: new disk, initializing [ 4360.363784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4360.388373] Lustre: 136206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4360.494415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4366.953194] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4370.030813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4370.045486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4370.172417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4376.490884] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4381.941584] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4382.106268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4387.247898] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4387.384329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4390.880479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4390.892046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.900095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4390.905775] Lustre: Skipped 1 previous similar message [ 4395.825688] Lustre: server umount lustre-OST0000 complete [ 4401.121662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.135996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.153735] Lustre: Skipped 1 previous similar message [ 4401.182641] Lustre: Skipped 2 previous similar messages [ 4406.246918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4406.254927] Lustre: Skipped 1 previous similar message [ 4406.890358] Lustre: server umount lustre-MDT0000 complete [ 4411.030298] LustreError: 134228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845488 with bad export cookie 15525510385384807945 [ 4411.044341] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4411.666433] Lustre: server umount lustre-MDT0001 complete [ 4421.282890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.824386] LustreError: 138424:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.833801] LustreError: 138424:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4421.970400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.782709] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4427.238921] LustreError: 138425:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.333268] LustreError: 138424:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.448635] LustreError: 138425:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.481799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.969508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4441.861827] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4451.427914] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4458.120519] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4465.047736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4465.462791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.643385] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4481.716256] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4487.402521] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4487.772245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4492.649848] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4492.900583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4504.329630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.596252] Lustre: lustre-OST0001: new disk, initializing [ 4504.600337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4504.609641] Lustre: 141797:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4504.721459] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4511.885581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4511.895217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4512.032288] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4512.436754] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4522.422892] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4522.746607] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4528.187758] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4528.454819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4536.208185] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e740b8bc000.ost_server_uuid 50 [ 4538.191768] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e740b8bc000.ost_server_uuid in IDLE state after 0 sec [ 4553.187229] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4553.194710] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4553.199966] LustreError: Skipped 1 previous similar message [ 4553.204692] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4553.214932] Lustre: Skipped 1 previous similar message [ 4555.446381] Lustre: server umount lustre-OST0001 complete [ 4558.319663] LustreError: 140243:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.335131] LustreError: 140243:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4563.427742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.438525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4563.459533] Lustre: Skipped 2 previous similar messages [ 4563.475780] Lustre: Skipped 1 previous similar message [ 4566.533972] Lustre: server umount lustre-OST0000 complete [ 4573.664629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.701043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4573.716965] Lustre: Skipped 1 previous similar message [ 4578.226942] Lustre: server umount lustre-MDT0000 complete [ 4578.786150] LustreError: 138424:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4578.804886] LustreError: 138424:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4582.457668] LustreError: 138405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845660 with bad export cookie 15525510385384808827 [ 4582.464325] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4582.468546] LustreError: 138405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4582.882970] Lustre: server umount lustre-MDT0001 complete [ 4606.733428] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4609.947199] Key type lgssc unregistered [ 4610.206626] LNet: 144805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4610.214495] LNetError: 144805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4610.228609] LNet: Removed LNI 192.168.202.154@tcp [ 4611.229719] Key type .llcrypt unregistered [ 4611.231458] Key type ._llcrypt unregistered [ 4660.255984] Key type ._llcrypt registered [ 4660.257608] Key type .llcrypt registered [ 4660.417275] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 4680.382408] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4681.892284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4681.965671] alg: No test for adler32 (adler32-zlib) [ 4682.973937] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 4683.262354] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4684.999209] Key type lgssc registered [ 4686.650661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4730.332188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4730.367202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.923077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4732.000355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4732.104521] Lustre: lustre-MDT0000: new disk, initializing [ 4732.186212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4732.205661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4738.061400] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4750.291668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.445285] Lustre: 148966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4750.494443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4750.497888] Lustre: Skipped 1 previous similar message [ 4750.555536] Lustre: lustre-MDT0001: new disk, initializing [ 4750.629218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4750.664445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4750.673286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4756.498886] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4768.214418] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4775.833836] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4785.247790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4785.538458] Lustre: lustre-OST0000: new disk, initializing [ 4785.541832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4785.549710] Lustre: 150937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4785.643708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.423061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4792.443689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4792.620059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4793.242440] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4805.137890] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4811.165532] Lustre: Failing over lustre-OST0000 [ 4811.309633] Lustre: server umount lustre-OST0000 complete [ 4828.639189] Lustre: 147190:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845890/real 1787845890] req@00000000e42233c6 x1874692156722944/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787845906 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4828.672209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.664376] Lustre: 147191:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845891/real 1787845891] req@0000000070aab9b6 x1874692156724096/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787845907 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4829.691776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.946142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.406903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4831.446706] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4834.463496] Lustre: 147191:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845896/real 1787845896] req@00000000771c3282 x1874692156725120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787845912 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4834.494323] Lustre: 147191:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4838.573416] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4838.658881] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4839.583155] Lustre: 147193:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845901/real 1787845901] req@000000005f13fb3f x1874692156726272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787845917 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4839.740522] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4839.744289] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4843.568918] Lustre: Failing over lustre-MDT0000 [ 4843.698502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4843.715555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4843.716341] Lustre: Skipped 2 previous similar messages [ 4843.724571] Lustre: Skipped 2 previous similar messages [ 4843.924449] Lustre: server umount lustre-MDT0000 complete [ 4844.514216] Lustre: 147193:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845906/real 1787845906] req@0000000010c2bd07 x1874692156727424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787845922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4844.545751] Lustre: 147193:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4848.901790] LustreError: 148973:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.919796] LustreError: 148973:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4854.035034] LustreError: 151237:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.060529] LustreError: 151237:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4859.138885] LustreError: 148974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.158768] LustreError: 148974:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4864.263947] LustreError: 148975:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.292069] LustreError: 148975:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4864.417785] Lustre: 147192:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845926/real 1787845926] req@0000000059d976ff x1874692156740224/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787845942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4864.458558] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.898820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4869.377945] LustreError: 148974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4869.408156] LustreError: 148974:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4874.721313] LustreError: 147189:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004bcc6950 x1874692156744576/t0(0) o250->MGC192.168.202.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4875.060980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4875.136518] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4879.619985] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4880.395696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4880.405057] Lustre: Skipped 1 previous similar message [ 4880.434900] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4880.506172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4880.909420] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 4891.109897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4891.111349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.116300] LustreError: Skipped 1 previous similar message [ 4891.117946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4891.152916] Lustre: Skipped 1 previous similar message [ 4895.721823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4895.734533] Lustre: Skipped 2 previous similar messages [ 4897.138556] Lustre: server umount lustre-OST0000 complete [ 4905.954039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4905.955812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4905.965308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4905.985721] Lustre: Skipped 1 previous similar message [ 4907.894637] Lustre: server umount lustre-MDT0000 complete [ 4911.080471] LustreError: 150099:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.108493] LustreError: 150099:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 4912.747275] LustreError: 150936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845990 with bad export cookie 15287140302150702165 [ 4912.772659] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4912.776698] LustreError: 150936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4913.502802] Lustre: server umount lustre-MDT0001 complete [ 4926.045961] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4929.267321] Key type lgssc unregistered [ 4929.691455] LNet: 154603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4929.704197] LNetError: 154603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4930.729302] LNet: Removed LNI 192.168.202.154@tcp [ 4932.340121] Key type .llcrypt unregistered [ 4932.344362] Key type ._llcrypt unregistered [ 4976.672549] Key type ._llcrypt registered [ 4976.677761] Key type .llcrypt registered [ 4976.804576] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4978.273651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4978.389305] alg: No test for adler32 (adler32-zlib) [ 4979.490557] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 4979.721255] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4981.439369] Key type lgssc registered [ 4982.800574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4991.899865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4991.917609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.463191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.242307] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5003.751712] LustreError: 156125:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.790265] LustreError: 156125:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5008.867158] LustreError: 156124:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5010.498109] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5013.997951] LustreError: 156125:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.247122] LustreError: 156153:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5015.264434] LustreError: 156153:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5015.294417] LustreError: 156153:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 5015.876104] Lustre: server umount lustre-MDT0000 complete [ 5033.862806] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5047.156460] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5089.642896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.951474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5090.038552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5090.127877] Lustre: lustre-MDT0000: new disk, initializing [ 5090.210794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5090.247946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5094.796190] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5105.562701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.716224] Lustre: 159459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5105.826196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5105.836731] Lustre: Skipped 1 previous similar message [ 5105.923051] Lustre: lustre-MDT0001: new disk, initializing [ 5106.067693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5106.108334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5106.120190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5111.511765] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5122.633935] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5129.525708] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5137.611332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5137.974912] Lustre: lustre-OST0000: new disk, initializing [ 5137.977411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5137.982360] Lustre: 161429:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5138.106259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5144.177311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5144.190528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5144.316750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5145.216564] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5156.873777] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5161.610417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5163.943473] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:57:19 (1787846239) [ 5444.084290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5444.097127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.112092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5446.627956] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.629013] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5446.639326] Lustre: Skipped 1 previous similar message [ 5446.653352] Lustre: Skipped 2 previous similar messages [ 5450.577300] Lustre: server umount lustre-MDT0000 complete [ 5451.749598] LustreError: 161015:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5451.774943] LustreError: 161015:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5454.528202] LustreError: 159453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846532 with bad export cookie 11872362509792231403 [ 5454.537888] LustreError: 159453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5454.549795] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5456.863705] LustreError: 161437:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.867278] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.894347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5459.872733] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5461.048746] Lustre: server umount lustre-MDT0001 complete [ 5471.976641] Lustre: server umount lustre-OST0000 complete [ 5482.599709] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5493.643520] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5536.390142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.676183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5536.711722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5536.768280] Lustre: lustre-MDT0000: new disk, initializing [ 5536.833980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5536.854682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5541.502330] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5553.565047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.771183] Lustre: 166113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5553.850268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5553.861506] Lustre: Skipped 1 previous similar message [ 5553.980296] Lustre: lustre-MDT0001: new disk, initializing [ 5554.087463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5554.119617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5554.133711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5559.275912] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5570.287518] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5577.306577] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5584.191551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5584.434492] Lustre: lustre-OST0000: new disk, initializing [ 5584.437676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5584.443556] Lustre: 168080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5584.517941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5590.084512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5590.092669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5590.209899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5591.060467] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5601.472824] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5606.696511] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5606.940335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5612.589927] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5612.785952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5616.097589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.107711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5616.120622] Lustre: Skipped 1 previous similar message [ 5616.145357] Lustre: Skipped 1 previous similar message [ 5621.866442] Lustre: server umount lustre-OST0000 complete [ 5626.336859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.341885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5626.351707] Lustre: Skipped 2 previous similar messages [ 5626.352341] Lustre: Skipped 1 previous similar message [ 5632.635992] Lustre: server umount lustre-MDT0000 complete [ 5636.590839] LustreError: 167330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5636.602132] LustreError: 167330:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5637.169282] LustreError: 168079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846714 with bad export cookie 11872362509792390233 [ 5637.179964] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5637.183615] LustreError: 168079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5637.634666] Lustre: server umount lustre-MDT0001 complete [ 5648.789662] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:05:24 (1787846724) [ 5668.896831] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5680.248632] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5723.916423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5724.063903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5724.081430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5724.127347] Lustre: lustre-MDT0000: new disk, initializing [ 5724.226140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.237923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5731.915960] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5746.969696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.153481] Lustre: 172839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5747.694440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5754.301260] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5766.214591] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5774.301482] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5783.865234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5784.224335] Lustre: lustre-OST0000: new disk, initializing [ 5784.230218] Lustre: Skipped 1 previous similar message [ 5784.235529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5784.243135] Lustre: Skipped 2 previous similar messages [ 5784.250106] Lustre: 174820:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5791.338249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5791.353734] Lustre: Skipped 1 previous similar message [ 5791.363836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5791.500875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5792.574866] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5803.775816] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5817.314952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5817.316188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.320155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5817.320165] Lustre: Skipped 3 previous similar messages [ 5817.330710] Lustre: Skipped 1 previous similar message [ 5819.964993] Lustre: server umount lustre-OST0000 complete [ 5824.481271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5824.495374] LustreError: Skipped 1 previous similar message [ 5830.951799] Lustre: server umount lustre-MDT0000 complete [ 5832.675446] LustreError: 173930:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.693521] LustreError: 173930:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5835.303673] LustreError: 172833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846913 with bad export cookie 11872362509792391115 [ 5835.311313] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5835.324057] LustreError: 172833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5835.597930] Lustre: server umount lustre-MDT0001 complete [ 5846.123680] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:08:42 (1787846922) [ 5867.357637] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 5884.990786] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5928.431604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.034564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5929.160922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5929.312534] Lustre: lustre-MDT0000: new disk, initializing [ 5929.435669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5929.445185] Lustre: Skipped 2 previous similar messages [ 5929.460816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5935.554499] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5946.688918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.824670] Lustre: 179167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5947.031048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5952.633665] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5964.120842] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5970.391268] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5978.356163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5978.716966] Lustre: 181142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5984.315255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5984.511873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5986.055380] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 5996.352978] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6010.339372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.349260] Lustre: Skipped 2 previous similar messages [ 6010.352817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6010.375384] LustreError: 181438:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.395045] LustreError: 181438:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6010.583379] Lustre: server umount lustre-OST0000 complete [ 6015.462248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6015.476332] Lustre: Skipped 4 previous similar messages [ 6021.493899] Lustre: server umount lustre-MDT0000 complete [ 6025.551200] LustreError: 179157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847103 with bad export cookie 11872362509792392200 [ 6025.555913] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6025.558835] LustreError: 179157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6026.155200] Lustre: server umount lustre-MDT0001 complete [ 6040.321574] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 6043.979505] Key type lgssc unregistered [ 6044.322951] LNet: 183005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6044.332544] LNetError: 183005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6045.354317] LNet: Removed LNI 192.168.202.154@tcp [ 6046.650617] Key type .llcrypt unregistered [ 6046.656067] Key type ._llcrypt unregistered [ 6073.535312] Key type ._llcrypt registered [ 6073.536969] Key type .llcrypt registered [ 6073.702396] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6074.671330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6074.779794] alg: No test for adler32 (adler32-zlib) [ 6075.995473] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 6076.303498] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 6078.047558] Key type lgssc registered [ 6079.204771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6088.746876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6088.788796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.478540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6097.719416] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6100.962166] LustreError: 184423:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.986078] LustreError: 184423:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6106.091050] LustreError: 184424:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.958259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6109.772590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6115.819864] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6123.686412] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6129.453642] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6137.221654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6137.623548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.200943] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6152.027959] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6161.155899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6180.832586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6180.838194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.862679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6181.860790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6181.861461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6181.874091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6187.046919] Lustre: server umount lustre-OST0000 complete [ 6191.584433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6191.594854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.628504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6191.640475] Lustre: Skipped 1 previous similar message [ 6197.218558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6197.223264] Lustre: Skipped 3 previous similar messages [ 6197.722837] Lustre: server umount lustre-MDT0000 complete [ 6201.948966] LustreError: 184404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847279 with bad export cookie 2576778764078151818 [ 6201.960988] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6202.385843] Lustre: server umount lustre-MDT0001 complete [ 6214.454191] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 6218.580637] Key type lgssc unregistered [ 6219.022250] LNet: 187986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6219.025622] LNetError: 187986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6219.046311] LNet: Removed LNI 192.168.202.154@tcp [ 6220.214178] Key type .llcrypt unregistered [ 6220.216251] Key type ._llcrypt unregistered [ 6248.887613] Key type ._llcrypt registered [ 6248.889909] Key type .llcrypt registered [ 6249.082914] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6250.217773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6250.228724] alg: No test for adler32 (adler32-zlib) [ 6251.436479] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 6251.774267] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 6253.479232] Key type lgssc registered [ 6254.865054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6263.245735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6263.258244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.762970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6269.526439] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6275.045058] LustreError: 189404:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6275.072410] LustreError: 189404:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6278.756479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.390627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6285.127277] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6294.169894] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6301.402975] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6309.585158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6310.013682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6317.349979] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6327.072425] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6334.611074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6345.191861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6345.196252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6345.204909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6345.220537] Lustre: Skipped 1 previous similar message [ 6349.591628] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 6350.876664] Lustre: server umount lustre-OST0000 complete [ 6359.294585] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8e7406f9d800.ost_server_uuid 50 [ 6361.012511] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6374.542531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.545286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6374.554156] Lustre: Skipped 2 previous similar messages [ 6374.568860] Lustre: Skipped 3 previous similar messages [ 6380.937023] Lustre: server umount lustre-MDT0000 complete [ 6381.029497] LustreError: 192007:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.043735] LustreError: 192007:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6384.741897] LustreError: 189386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847462 with bad export cookie 7328106903910852169 [ 6384.745293] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6384.749050] LustreError: 189386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6385.553465] Lustre: server umount lustre-MDT0001 complete [ 6394.716526] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:17:50 (1787847470) [ 6402.436106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6403.105141] LustreError: 193343:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6403.204539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6407.885538] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6408.675730] LustreError: 193344:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.775886] LustreError: 193343:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6417.707938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6417.887961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6417.890853] Lustre: Skipped 2 previous similar messages [ 6417.994305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6422.882820] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6431.646104] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6438.179322] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6445.573245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6445.964362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.053906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6452.521586] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6460.035652] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6466.835970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6467.033276] Lustre: lustre-OST0001: new disk, initializing [ 6467.036478] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6467.042643] Lustre: 196185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6467.135461] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6473.397383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6473.410482] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6473.501109] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6473.758929] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6483.630865] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6488.034543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6488.046479] LustreError: Skipped 1 previous similar message [ 6488.049912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6488.068097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6489.057501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6499.301895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6499.316639] Lustre: Skipped 7 previous similar messages [ 6501.343912] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6501.448627] Lustre: server umount lustre-OST0000 complete [ 6504.418742] LustreError: 195168:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6504.467925] LustreError: 195168:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6509.400853] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6509.781665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6517.599607] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6518.076055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6524.913406] LustreError: 196194:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.982180] LustreError: 196194:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 6535.136782] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6535.149868] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.152081] LustreError: Skipped 1 previous similar message [ 6535.153219] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6535.153225] Lustre: Skipped 1 previous similar message [ 6535.193340] Lustre: Skipped 2 previous similar messages [ 6540.482216] Lustre: server umount lustre-OST0001 complete [ 6545.890980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6545.921778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.218304] Lustre: server umount lustre-MDT0000 complete [ 6556.933302] LustreError: 193325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847634 with bad export cookie 7328106903910853240 [ 6556.937467] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6556.943332] LustreError: 193325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6557.459876] Lustre: server umount lustre-MDT0001 complete [ 6583.544598] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:20:59 (1787847659) [ 6589.760483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.036217] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6590.069956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6590.233839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.874402] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6603.690041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.825218] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6603.897194] Lustre: 199629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6603.989052] LustreError: 199641:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.011872] LustreError: 199641:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 6609.141209] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6619.739512] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6627.656395] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6636.493564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.710104] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6644.050881] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6644.388519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6652.998772] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6661.179534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.276703] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6661.492429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6661.504403] Lustre: Skipped 2 previous similar messages [ 6669.010974] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6677.365963] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6680.545718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6680.549059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.566227] Lustre: Skipped 1 previous similar message [ 6680.571577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6680.575898] Lustre: Skipped 4 previous similar messages [ 6686.538523] Lustre: server umount lustre-OST0000 complete [ 6688.517219] LustreError: 201481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.127890] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6705.134027] LustreError: Skipped 1 previous similar message [ 6705.140911] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6705.156879] Lustre: Skipped 1 previous similar message [ 6708.733664] Lustre: server umount lustre-OST0001 complete [ 6721.403426] Lustre: server umount lustre-MDT0000 complete [ 6726.318821] LustreError: 199623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847804 with bad export cookie 7328106903910854647 [ 6726.322283] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.350474] LustreError: 199623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6727.075896] Lustre: server umount lustre-MDT0001 complete [ 6756.365367] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:23:52 (1787847832) [ 6774.985655] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 6787.767947] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6828.123984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.471991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6828.537082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6828.680856] Lustre: lustre-MDT0000: new disk, initializing [ 6828.801250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6828.825851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6835.079763] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6850.609995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.799733] Lustre: 207731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6850.927791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6850.937702] Lustre: Skipped 1 previous similar message [ 6851.035110] Lustre: lustre-MDT0001: new disk, initializing [ 6851.259481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6851.277667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6857.451616] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6869.076388] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6876.943329] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6884.148704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.502309] Lustre: lustre-OST0000: new disk, initializing [ 6884.507487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6884.514439] Lustre: 209702:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6892.166638] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6893.654143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6893.668489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6893.904665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6903.699165] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6909.337401] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6909.578676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6915.290467] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6915.650897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6919.651154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6919.662703] LustreError: Skipped 1 previous similar message [ 6919.674063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.679537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6919.686527] Lustre: Skipped 4 previous similar messages [ 6919.695093] Lustre: Skipped 8 previous similar messages [ 6924.774538] LustreError: 209688:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.792633] LustreError: 209688:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [ 6924.950509] Lustre: server umount lustre-OST0000 complete [ 6936.782712] Lustre: server umount lustre-MDT0000 complete [ 6942.018624] LustreError: 207724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848019 with bad export cookie 7328106903910856208 [ 6942.044585] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6942.580462] Lustre: server umount lustre-MDT0001 complete [ 6954.155244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.553907] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6971.975669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.298809] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 6989.045402] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6995.928193] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7005.062562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7013.441713] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7023.116276] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7029.704274] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7030.006881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7035.726406] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7036.025221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7038.432225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7038.438571] LustreError: Skipped 2 previous similar messages [ 7038.444995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.459687] Lustre: Skipped 2 previous similar messages [ 7044.563382] Lustre: server umount lustre-OST0000 complete [ 7053.473882] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7060.043453] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7091.661609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7092.102988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.105925] Lustre: Skipped 5 previous similar messages [ 7100.282737] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7105.383171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7105.415712] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7109.648741] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7111.678660] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7115.688418] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7115.699166] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7120.864835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7120.879348] Lustre: Skipped 7 previous similar messages [ 7122.530517] Lustre: server umount lustre-OST0000 complete [ 7141.345766] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7141.789559] Lustre: server umount lustre-MDT0000 complete [ 7145.783031] LustreError: 211903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848223 with bad export cookie 7328106903910857090 [ 7145.785652] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7145.796861] LustreError: 211903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7146.173091] Lustre: server umount lustre-MDT0001 complete [ 7155.641557] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:30:31 (1787848231) [ 7162.822033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7168.425449] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7178.789066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.818211] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7194.809889] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7201.588529] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7208.624388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7214.648226] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7224.851192] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7230.133366] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7230.386340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7236.860573] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7237.181242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7244.998392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7245.214965] Lustre: lustre-OST0001: new disk, initializing [ 7245.219955] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7245.226903] Lustre: 221266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7252.117693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7252.126282] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7252.267326] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7253.705269] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7264.996103] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7270.842967] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7271.089105] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7277.471513] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7277.669424] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7283.174861] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.177422] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7283.183062] Lustre: Skipped 5 previous similar messages [ 7283.196062] LustreError: Skipped 3 previous similar messages [ 7286.402364] Lustre: server umount lustre-OST0001 complete [ 7288.299339] LustreError: 219719:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.321296] LustreError: 219719:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 7293.852806] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7294.098375] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7299.919174] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7300.084235] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7333.647975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7339.000276] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7339.018475] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7339.022042] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7339.047281] Lustre: Skipped 1 previous similar message [ 7341.982219] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7353.096885] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7355.447272] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7364.196376] Lustre: server umount lustre-OST0001 complete [ 7385.064081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7385.070234] Lustre: Skipped 10 previous similar messages [ 7389.713173] Lustre: server umount lustre-OST0000 complete [ 7400.913765] Lustre: server umount lustre-MDT0000 complete [ 7406.007719] LustreError: 218630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848483 with bad export cookie 7328106903910858056 [ 7406.026211] LustreError: 218630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7406.031854] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7420.384474] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7420.734812] Lustre: server umount lustre-MDT0001 complete [ 7453.123923] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:35:29 (1787848529) [ 7461.186757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.713882] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7461.737386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7467.365291] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7477.880656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.061261] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7478.115902] Lustre: 226516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7484.235492] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7493.995926] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7502.647727] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7512.252611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.493469] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7522.054296] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7532.659878] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7544.307631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.494433] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7553.537269] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7563.665498] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7565.909347] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7571.760663] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7572.071722] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7578.752389] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7579.168442] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7588.135809] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e74092b7000.ost_server_uuid 50 [ 7590.069894] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e74092b7000.ost_server_uuid in IDLE state after 0 sec [ 7591.384963] Lustre: Permanently deactivating lustre-OST0001 [ 7591.419417] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7596.278338] Lustre: Permanently reactivating lustre-OST0001 [ 7596.300439] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7596.307501] Lustre: Skipped 2 previous similar messages [ 7601.643694] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7601.649022] LustreError: Skipped 3 previous similar messages [ 7601.653095] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.663982] Lustre: Skipped 7 previous similar messages [ 7607.769221] Lustre: server umount lustre-OST0001 complete [ 7619.004355] Lustre: server umount lustre-OST0000 complete [ 7630.766624] Lustre: server umount lustre-MDT0000 complete [ 7635.363266] LustreError: 226508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848713 with bad export cookie 7328106903910859344 [ 7635.364930] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.374702] LustreError: 226508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7635.995982] Lustre: server umount lustre-MDT0001 complete [ 7666.765605] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:39:02 (1787848742) [ 7677.495873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.766233] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7677.791482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7677.800423] Lustre: Skipped 2 previous similar messages [ 7678.033451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7678.040065] Lustre: Skipped 9 previous similar messages [ 7683.753643] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7695.442181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.530673] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7695.579427] Lustre: 233218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7701.934038] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7711.860865] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7720.874129] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7730.740134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7731.001799] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7731.043405] Lustre: Permanently deactivating lustre-OST0000 [ 7731.061043] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7739.979748] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7740.696885] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7751.554163] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7760.160257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7760.350397] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7767.775620] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7769.381801] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7778.174733] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7780.346462] Lustre: Permanently reactivating lustre-OST0000 [ 7780.369859] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7780.383380] Lustre: Skipped 2 previous similar messages [ 7785.622893] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7785.640277] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 46 seconds. I think it's dead, and I am evicting it. exp 00000000b723517f, cur 1787848863 deadline 1787848847 last 1787848817 [ 7785.725267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7802.913727] Lustre: server umount lustre-OST0001 complete [ 7803.894216] LustreError: 236406:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7803.909556] LustreError: 236406:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 43 previous similar messages [ 7822.303289] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7822.574303] Lustre: server umount lustre-OST0000 complete [ 7833.937887] Lustre: server umount lustre-MDT0000 complete [ 7838.712365] LustreError: 233209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848916 with bad export cookie 7328106903910861010 [ 7838.720798] LustreError: 233209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7838.721449] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7839.114821] Lustre: server umount lustre-MDT0001 complete [ 7851.303648] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 7854.346585] Key type lgssc unregistered [ 7854.601595] LNet: 238111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7854.606190] LNetError: 238111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7854.619548] LNet: Removed LNI 192.168.202.154@tcp [ 7855.579982] Key type .llcrypt unregistered [ 7855.584179] Key type ._llcrypt unregistered [ 7881.708966] Key type ._llcrypt registered [ 7881.712265] Key type .llcrypt registered [ 7881.876771] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7883.589442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7883.637569] alg: No test for adler32 (adler32-zlib) [ 7884.959710] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 7885.411419] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 7887.231396] Key type lgssc registered [ 7888.932107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7902.231772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7902.280893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.975207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7909.929066] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7913.443449] LustreError: 239580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.465069] LustreError: 239580:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7918.565057] LustreError: 239579:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7922.171372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7922.359529] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7922.398869] Lustre: 239568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7922.410973] Lustre: Permanently deactivating lustre-MDT0001 [ 7922.419234] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7922.794648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7929.423760] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7931.930044] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7939.447682] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7947.023547] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7956.907559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7957.364529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7965.499075] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 7967.605637] LustreError: 241421:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7967.647147] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7972.838156] LustreError: 241421:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7974.910317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7975.162166] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7980.028078] LustreError: 241421:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7980.056887] LustreError: 241421:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7984.040423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7984.391085] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7989.781857] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7989.799278] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7994.056709] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8003.779607] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8008.325388] Lustre: Permanently reactivating lustre-MDT0001 [ 8008.366815] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8008.379734] Lustre: Skipped 2 previous similar messages [ 8013.142127] Lustre: lustre-MDT0001: Client d1e74e10-c05c-400a-b6fc-56d6bfd6156d (at 192.168.202.54@tcp) reconnecting [ 8018.052126] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.058642] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 8018.064858] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8018.073777] LustreError: 243264:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 8018.079473] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8018.081652] LustreError: 243264:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 8018.081893] LustreError: 243264:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.154@tcp: cfg command failed: rc = -108 [ 8018.081943] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 8018.081943] [ 8018.083567] LustreError: 239572:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 8020.722545] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8022.730373] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8029.847724] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8030.220963] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8035.902886] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8036.408814] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8040.104157] Lustre: Permanently deactivating lustre-MDT0001 [ 8040.162912] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8040.186131] Lustre: Skipped 2 previous similar messages [ 8048.236758] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8055.778066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8055.782526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.789489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.308302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8059.016855] Lustre: server umount lustre-MDT0000 complete [ 8061.427360] LustreError: 239580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8061.457988] LustreError: 239580:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8063.227742] LustreError: 239559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849141 with bad export cookie 3945185889624089935 [ 8063.230296] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8063.232644] LustreError: 239559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8063.461623] Lustre: server umount lustre-MDT0001 complete [ 8070.077137] Lustre: server umount lustre-OST0000 complete [ 8084.517100] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:46:00 (1787849160) [ 8098.678218] Lustre: server umount lustre-OST0001 complete [ 8107.237443] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 8118.761884] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8164.915311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8165.168609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8165.178352] Lustre: Skipped 2 previous similar messages [ 8165.207517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8165.275439] Lustre: lustre-MDT0000: new disk, initializing [ 8165.385080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8165.400723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8170.565768] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8184.197676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.424614] Lustre: 247843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8184.711629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8184.725757] Lustre: Skipped 1 previous similar message [ 8184.870665] Lustre: lustre-MDT0001: new disk, initializing [ 8185.100404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8185.173884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8185.187825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8192.053888] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8204.646254] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8211.545384] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8219.762410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8220.235698] Lustre: lustre-OST0000: new disk, initializing [ 8220.238671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8220.248887] Lustre: 249815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8220.388068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8230.533465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8230.549666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8230.769548] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8230.779099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8243.035575] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8245.648320] LustreError: 247853:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8255.743273] LustreError: 247853:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8255.782216] LustreError: 247851:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8256.172384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.533365] Lustre: lustre-OST0001: new disk, initializing [ 8256.543403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8256.552606] Lustre: 250932:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8256.696472] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8263.367463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8263.390278] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8263.544889] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8265.871281] LustreError: 247851:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8266.402210] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8279.315766] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8281.174739] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 8289.249754] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8289.258135] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.258473] LustreError: Skipped 1 previous similar message [ 8289.269383] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8289.277965] Lustre: Skipped 4 previous similar messages [ 8289.297030] Lustre: Skipped 4 previous similar messages [ 8290.565568] Lustre: server umount lustre-OST0001 complete [ 8294.371591] LustreError: 249798:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8309.734601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.763152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.852799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.860250] Lustre: Skipped 2 previous similar messages [ 8315.826989] Lustre: server umount lustre-OST0000 complete [ 8323.552295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8323.559952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.582099] Lustre: Skipped 1 previous similar message [ 8323.593185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8335.328182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8335.796909] Lustre: server umount lustre-MDT0000 complete [ 8339.425122] LustreError: 247853:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.443404] LustreError: 247853:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 8340.642595] LustreError: 250931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849418 with bad export cookie 3945185889624092749 [ 8340.653624] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8340.657747] LustreError: 250931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8347.233740] Lustre: server umount lustre-MDT0001 complete [ 8359.948844] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8363.861270] Key type lgssc unregistered [ 8364.227487] LNet: 252973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8364.239272] LNetError: 252973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8364.253484] LNet: Removed LNI 192.168.202.154@tcp [ 8365.441171] Key type .llcrypt unregistered [ 8365.442399] Key type ._llcrypt unregistered [ 8420.932153] Key type ._llcrypt registered [ 8420.933954] Key type .llcrypt registered [ 8421.007522] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8423.655977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8423.692834] alg: No test for adler32 (adler32-zlib) [ 8424.922462] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 8425.310262] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8427.191288] Key type lgssc registered [ 8430.226030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8443.172295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8443.195424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.612482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8444.654486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8444.996496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8452.963193] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8465.475431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8465.610503] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8465.700630] Lustre: 255182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8466.128957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8473.109433] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8482.880393] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8490.059618] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8498.542375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8498.823664] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8499.142549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8505.054481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8507.895497] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8516.789254] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8571.873262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8571.880536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.884227] LustreError: Skipped 1 previous similar message [ 8571.887672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8571.925850] Lustre: Skipped 1 previous similar message [ 8576.570221] Lustre: server umount lustre-OST0000 complete [ 8583.430601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8601.897738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8602.616380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.310514] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8611.817621] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8611.831178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8611.834185] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8611.858583] Lustre: Skipped 1 previous similar message [ 8620.215572] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8663.033617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8663.036599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.060162] LustreError: Skipped 1 previous similar message [ 8663.075550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8663.083601] Lustre: Skipped 1 previous similar message [ 8666.308422] Lustre: server umount lustre-OST0000 complete [ 8671.204149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8671.206881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.230748] Lustre: Skipped 1 previous similar message [ 8671.260527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8671.271796] Lustre: Skipped 1 previous similar message [ 8673.252973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8673.275621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8673.291156] Lustre: Skipped 1 previous similar message [ 8677.361397] Lustre: server umount lustre-MDT0000 complete [ 8681.630071] LustreError: 255177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849759 with bad export cookie 9354869822827257665 [ 8681.639408] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8682.058115] Lustre: server umount lustre-MDT0001 complete [ 8695.563032] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8698.949208] Key type lgssc unregistered [ 8699.363559] LNet: 260632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8699.373463] LNetError: 260632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8699.410220] LNet: Removed LNI 192.168.202.154@tcp [ 8700.439775] Key type .llcrypt unregistered [ 8700.441852] Key type ._llcrypt unregistered [ 8728.487027] Key type ._llcrypt registered [ 8728.489100] Key type .llcrypt registered [ 8728.631862] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8729.697161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8729.785833] alg: No test for adler32 (adler32-zlib) [ 8730.844578] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 8731.305806] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8733.008090] Key type lgssc registered [ 8734.054207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8743.624909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8743.665718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8745.390573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8751.399817] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8755.681914] LustreError: 262049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8755.697467] LustreError: 262049:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8760.811992] LustreError: 262050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8762.091697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.602862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8767.659199] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8776.238081] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8783.561631] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8792.253356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.781887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8800.031048] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8809.771497] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8815.428567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8841.191756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.200271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8841.213336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8841.234786] Lustre: Skipped 1 previous similar message [ 8846.318764] LustreError: 263871:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.336222] LustreError: 263871:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8846.440918] Lustre: server umount lustre-OST0000 complete [ 8851.426378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8851.441802] Lustre: Skipped 1 previous similar message [ 8851.445601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8851.451221] Lustre: Skipped 1 previous similar message [ 8856.544537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8856.551571] Lustre: Skipped 1 previous similar message [ 8857.515290] Lustre: server umount lustre-MDT0000 complete [ 8861.562432] LustreError: 262030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849939 with bad export cookie 8035288919818034018 [ 8861.571524] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8861.991949] Lustre: server umount lustre-MDT0001 complete [ 8877.087473] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 8880.906763] Key type lgssc unregistered [ 8881.241802] LNet: 266365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8881.249547] LNetError: 266365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8881.274886] LNet: Removed LNI 192.168.202.154@tcp [ 8882.282446] Key type .llcrypt unregistered [ 8882.286376] Key type ._llcrypt unregistered [ 8901.325632] Key type ._llcrypt registered [ 8901.329473] Key type .llcrypt registered [ 8901.541194] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 8903.698178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8903.752218] alg: No test for adler32 (adler32-zlib) [ 8904.810803] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 8905.194496] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 8906.911266] Key type lgssc registered [ 8908.048550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8918.064058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8918.078762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.789561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8924.660727] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8929.255302] LustreError: 267591:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8929.283213] LustreError: 267591:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8934.383408] LustreError: 267592:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.816490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.478940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8944.040909] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8953.832625] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8961.211112] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8969.324747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8969.593113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8976.713910] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 8985.668468] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8990.087853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9003.501166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9003.515860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.530474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9004.748788] Lustre: server umount lustre-OST0000 complete [ 9010.656579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9010.682677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9015.777681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9015.781828] Lustre: Skipped 1 previous similar message [ 9019.996027] Lustre: server umount lustre-MDT0000 complete [ 9020.902693] LustreError: 268319:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9020.938684] LustreError: 268319:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9024.526671] LustreError: 267571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850102 with bad export cookie 9840056929712261159 [ 9024.540157] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9024.541925] LustreError: 267571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9024.941330] Lustre: server umount lustre-MDT0001 complete [ 9039.218111] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9042.947109] Key type lgssc unregistered [ 9043.363724] LNet: 271283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9043.375677] LNetError: 271283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9043.406984] LNet: Removed LNI 192.168.202.154@tcp [ 9044.581134] Key type .llcrypt unregistered [ 9044.582859] Key type ._llcrypt unregistered [ 9073.738551] Key type ._llcrypt registered [ 9073.740867] Key type .llcrypt registered [ 9073.861777] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9075.148980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9075.259449] alg: No test for adler32 (adler32-zlib) [ 9076.398676] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 9076.721805] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9078.495748] Key type lgssc registered [ 9079.923495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9089.557751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9089.607789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9091.243706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9095.808659] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9101.798751] LustreError: 272703:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9101.816421] LustreError: 272703:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9105.010379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9105.500567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9111.016683] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9119.609407] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9126.772183] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9135.039475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9135.382489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9136.394400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9142.723338] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9150.889577] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9181.687419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9181.693818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9181.740204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9181.751264] Lustre: Skipped 1 previous similar message [ 9185.348709] Lustre: server umount lustre-OST0000 complete [ 9191.905079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9191.924268] Lustre: Skipped 1 previous similar message [ 9191.950170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9197.024841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9197.037117] Lustre: Skipped 2 previous similar messages [ 9197.994805] Lustre: server umount lustre-MDT0000 complete [ 9202.145887] LustreError: 273430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9202.170501] LustreError: 273430:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9202.714958] LustreError: 274546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850280 with bad export cookie 15350327305243880593 [ 9202.725378] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9203.161207] Lustre: server umount lustre-MDT0001 complete [ 9216.437813] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9219.846511] Key type lgssc unregistered [ 9220.223665] LNet: 277012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9220.235183] LNetError: 277012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9221.289712] LNet: Removed LNI 192.168.202.154@tcp [ 9222.306883] Key type .llcrypt unregistered [ 9222.309831] Key type ._llcrypt unregistered [ 9242.834465] Key type ._llcrypt registered [ 9242.836391] Key type .llcrypt registered [ 9243.061940] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9245.582836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9245.699387] alg: No test for adler32 (adler32-zlib) [ 9246.780260] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 9246.931151] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9248.615190] Key type lgssc registered [ 9249.855937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9262.192693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9262.236108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9264.252053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.304644] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9274.851093] LustreError: 278249:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9274.864437] LustreError: 278249:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9279.639315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9279.968130] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9280.180270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9284.657277] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9292.862877] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9299.806922] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9306.597545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9307.120522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9310.228033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9314.448852] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9324.769946] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9339.885093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.915805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9339.919285] Lustre: Skipped 1 previous similar message [ 9343.993552] Lustre: server umount lustre-OST0000 complete [ 9350.116723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9350.125867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9350.136576] Lustre: Skipped 2 previous similar messages [ 9350.152301] Lustre: Skipped 1 previous similar message [ 9354.903041] Lustre: server umount lustre-MDT0000 complete [ 9355.234753] LustreError: 278236:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9355.263694] LustreError: 278236:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9359.334388] LustreError: 278233:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9359.366533] LustreError: 278233:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9359.488470] LustreError: 278213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850437 with bad export cookie 18128358853978577896 [ 9359.507409] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9360.043550] Lustre: server umount lustre-MDT0001 complete [ 9373.158688] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9376.827043] Key type lgssc unregistered [ 9377.262634] LNet: 281953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9377.270644] LNetError: 281953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9378.348150] LNet: Removed LNI 192.168.202.154@tcp [ 9379.366734] Key type .llcrypt unregistered [ 9379.368408] Key type ._llcrypt unregistered [ 9393.563767] Key type ._llcrypt registered [ 9393.574729] Key type .llcrypt registered [ 9393.743994] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9413.888750] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 9430.870939] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9432.717718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9432.760474] alg: No test for adler32 (adler32-zlib) [ 9433.921739] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 9434.393914] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9436.215195] Key type lgssc registered [ 9437.573879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9478.626630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9478.697851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9480.168026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9480.221739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9480.414588] Lustre: lustre-MDT0000: new disk, initializing [ 9480.717250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9480.829283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9487.932197] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9501.069450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.202700] Lustre: 285510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9501.243647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9501.247837] Lustre: Skipped 1 previous similar message [ 9501.317627] Lustre: lustre-MDT0001: new disk, initializing [ 9501.404557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9501.464107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9501.469624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9505.887476] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9517.386760] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9523.669934] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9532.733228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9533.077319] Lustre: lustre-OST0000: new disk, initializing [ 9533.082671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9533.087439] Lustre: 287480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9533.211378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9538.664592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9538.689466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9538.857642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9541.233752] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9552.348708] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9558.516039] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9558.851206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9564.961428] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9565.278803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9569.764175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9569.781231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9569.791051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9569.803265] Lustre: Skipped 1 previous similar message [ 9574.501446] Lustre: server umount lustre-OST0000 complete [ 9580.014051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9580.027165] Lustre: Skipped 1 previous similar message [ 9580.030236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9580.051768] Lustre: Skipped 2 previous similar messages [ 9585.122794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9585.132897] Lustre: Skipped 1 previous similar message [ 9589.216924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.235236] Lustre: Skipped 1 previous similar message [ 9593.311186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9593.724626] Lustre: server umount lustre-MDT0000 complete [ 9598.055253] LustreError: 287479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850675 with bad export cookie 10565128382300517469 [ 9598.061400] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9598.070581] LustreError: 287479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9598.622745] Lustre: server umount lustre-MDT0001 complete [ 9608.633124] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:11:24 (1787850684) [ 9616.067672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9616.789640] LustreError: 289895:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9616.808403] LustreError: 289895:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9616.868282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9621.984773] LustreError: 289896:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9623.024827] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9627.104402] LustreError: 289895:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9632.226115] LustreError: 289896:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9633.718159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9634.169094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9639.284719] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9647.634197] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9656.893565] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9667.323247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9667.510282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9674.115505] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9682.216642] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9699.297327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9699.305424] LustreError: Skipped 1 previous similar message [ 9699.308443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9699.339581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9702.377764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9702.391637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9704.159597] Lustre: server umount lustre-OST0000 complete [ 9708.802096] Lustre: server umount lustre-MDT0000 complete [ 9711.072652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9711.083839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9711.105567] LustreError: 289895:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9713.003358] LustreError: 289877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850790 with bad export cookie 10565128382300518351 [ 9713.017370] LustreError: 289877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9713.019158] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9713.388820] Lustre: server umount lustre-MDT0001 complete [ 9726.874805] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 9730.015603] Key type lgssc unregistered [ 9730.331906] LNet: 293458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9730.336534] LNetError: 293458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9730.355363] LNet: Removed LNI 192.168.202.154@tcp [ 9731.310088] Key type .llcrypt unregistered [ 9731.312907] Key type ._llcrypt unregistered [ 9767.121231] Key type ._llcrypt registered [ 9767.123374] Key type .llcrypt registered [ 9767.233925] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 9786.652939] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 9787.924965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9788.174000] alg: No test for adler32 (adler32-zlib) [ 9789.526827] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [ 9789.821112] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 9791.575308] Key type lgssc registered [ 9793.129787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9847.603797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9847.688124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9848.952112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9848.984074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9849.064868] Lustre: lustre-MDT0000: new disk, initializing [ 9849.155223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9849.186900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9854.485769] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9866.376537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.509411] Lustre: 297287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9866.552642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9866.555525] Lustre: Skipped 1 previous similar message [ 9866.606944] Lustre: lustre-MDT0001: new disk, initializing [ 9866.688168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9866.727785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9866.747226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9872.115170] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9883.889343] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9892.239769] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9902.418912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9902.746782] Lustre: lustre-OST2710: new disk, initializing [ 9902.750367] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9902.759420] Lustre: 299256:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9902.944708] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9909.291514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9909.311669] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9909.392978] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9911.008979] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9921.570028] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9929.572609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9929.858415] Lustre: lustre-OST03e8: new disk, initializing [ 9929.865403] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9929.878919] Lustre: 300322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9930.004861] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9936.846712] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [ 9939.079598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9939.088163] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9939.264260] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9944.539059] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9952.630047] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9952.884595] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9957.787797] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9958.022164] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9963.813675] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9964.066338] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9968.377660] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9968.614190] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9974.250150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9974.259151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9974.289221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9975.268314] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9976.491651] Lustre: server umount lustre-MDT0000 complete [ 9980.389172] LustreError: 298409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9980.413803] LustreError: 298409:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 9980.886843] LustreError: 297278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851058 with bad export cookie 334896923351234353 [ 9980.904174] LustreError: 297278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9980.905995] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9981.296980] Lustre: server umount lustre-MDT0001 complete [ 9996.371452] Lustre: server umount lustre-OST2710 complete [10011.988666] Lustre: server umount lustre-OST03e8 complete [10034.230392] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [10044.616478] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10085.655681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10086.022682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10086.047356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10086.182305] Lustre: lustre-MDT0000: new disk, initializing [10086.335144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10086.350464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10090.880416] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10102.934696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.062460] Lustre: 305574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10103.144320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10103.147902] Lustre: Skipped 1 previous similar message [10103.210958] Lustre: lustre-MDT0001: new disk, initializing [10103.353355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10103.394651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10103.416134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10109.556612] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10120.276693] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10127.452119] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10134.793175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10135.325245] Lustre: lustre-OST0000: new disk, initializing [10135.329813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10135.340116] Lustre: 307540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10135.417445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10141.820347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10141.828522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10141.979419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10143.501638] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10157.123845] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10162.553387] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10162.898248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10168.029447] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10168.272616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10172.896475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10172.905255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10172.915478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10172.926083] Lustre: Skipped 3 previous similar messages [10172.949758] Lustre: Skipped 1 previous similar message [10178.020646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10178.023460] Lustre: Skipped 1 previous similar message [10183.146086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10183.153202] Lustre: Skipped 1 previous similar message [10185.183422] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10185.326759] Lustre: server umount lustre-OST0000 complete [10190.305137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10190.309338] LustreError: Skipped 1 previous similar message [10190.312915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10190.322789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10195.761836] Lustre: server umount lustre-MDT0000 complete [10198.498351] LustreError: 305582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10198.526600] LustreError: 305582:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10200.135365] LustreError: 305565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851277 with bad export cookie 334896923351236026 [10200.139353] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10200.159371] LustreError: 305565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10200.650823] Lustre: server umount lustre-MDT0001 complete [10210.260413] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:21:26 (1787851286) [10211.802474] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10213.567786] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:21:29 (1787851289) [10217.737986] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10249.966200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10250.257582] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10250.289376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10250.474472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10256.139848] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10266.500620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10266.683475] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10266.802843] Lustre: 311405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10272.748507] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10295.596417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10295.910394] Lustre: Denying initial registration attempt from nid 192.168.202.154@tcp, specified as failover [10295.916423] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10295.919397] LustreError: 312859:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10295.943164] LustreError: 312859:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10295.948832] LustreError: 312859:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10296.080709] Lustre: server umount lustre-OST0000 complete [10296.086290] LustreError: 312859:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10307.556315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10307.560644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10307.571661] Lustre: Skipped 2 previous similar messages [10307.592403] Lustre: Skipped 3 previous similar messages [10311.577919] Lustre: server umount lustre-MDT0000 complete [10312.676446] LustreError: 311415:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10312.692823] LustreError: 311415:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10316.249830] LustreError: 311396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851394 with bad export cookie 334896923351236915 [10316.268149] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10316.270519] LustreError: 311396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10316.937920] Lustre: server umount lustre-MDT0001 complete [10335.312792] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10339.658875] Key type lgssc unregistered [10340.160339] LNet: 313986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10340.165582] LNetError: 313986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10341.221486] LNet: Removed LNI 192.168.202.154@tcp [10341.971161] Key type .llcrypt unregistered [10341.972990] Key type ._llcrypt unregistered [10356.086886] Key type ._llcrypt registered [10356.089897] Key type .llcrypt registered [10356.176743] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10357.668732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10357.762505] alg: No test for adler32 (adler32-zlib) [10358.785895] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [10358.995689] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10360.688346] Key type lgssc registered [10362.603482] Lustre: Echo OBD driver; http://www.lustre.org/ [10412.711132] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10423.982690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10424.005297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.436932] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10425.463374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10425.692086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10430.664273] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10440.774461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10440.899130] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10440.950686] Lustre: 316755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10441.158769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10446.456337] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10454.849866] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10460.956620] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10483.555377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10483.734269] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10483.963249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10492.031607] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10500.971458] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10507.745730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10507.762845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10507.782510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10508.263484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10508.264152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.301688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10508.309833] Lustre: Skipped 1 previous similar message [10509.730436] Lustre: server umount lustre-OST0000 complete [10517.989336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10517.993473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10518.006520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10521.380669] Lustre: server umount lustre-MDT0000 complete [10526.195884] LustreError: 318620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851604 with bad export cookie 889680884551553982 [10526.201524] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10526.206363] LustreError: 318620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10526.812521] Lustre: server umount lustre-MDT0001 complete [10539.303902] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10542.417385] Key type lgssc unregistered [10542.665774] LNet: 320332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10542.672184] LNetError: 320332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10542.684923] LNet: Removed LNI 192.168.202.154@tcp [10543.663550] Key type .llcrypt unregistered [10543.668411] Key type ._llcrypt unregistered [10570.881794] Key type ._llcrypt registered [10570.885980] Key type .llcrypt registered [10571.096667] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10572.705069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10572.754415] alg: No test for adler32 (adler32-zlib) [10573.997457] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [10574.391745] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10576.135210] Key type lgssc registered [10577.935192] Lustre: Echo OBD driver; http://www.lustre.org/ [10588.760288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10588.804625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10590.609155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10596.826397] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10600.937929] LustreError: 321753:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.976702] LustreError: 321753:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10606.049530] LustreError: 321752:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.179637] LustreError: 321753:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.437239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.926441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10618.208948] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10628.713786] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10635.330823] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10645.481613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10646.018810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10657.164955] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10665.494765] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10675.684163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10675.692491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10675.715772] Lustre: Skipped 1 previous similar message [10677.233796] Lustre: server umount lustre-MDT0000 complete [10678.054252] LustreError: 321748:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10681.849655] LustreError: 321732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851759 with bad export cookie 5241368891586473813 [10681.874312] LustreError: 321732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10681.876385] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10683.138839] LustreError: 324051:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.155132] LustreError: 324051:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10683.162251] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [10683.180136] Lustre: Skipped 2 previous similar messages [10685.942183] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10685.957306] Lustre: Skipped 1 previous similar message [10685.975435] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10688.454864] Lustre: server umount lustre-MDT0001 complete [10696.974921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10707.426726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10726.882296] LustreError: 321061:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005fa5801b x1874698333883520/t0(0) o250->MGC192.168.202.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10727.384292] LustreError: 323586:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10727.403654] LustreError: 323586:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10727.503448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10727.581103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10733.653863] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10741.752152] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10743.563588] LustreError: 325350:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10743.589185] LustreError: 325350:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [10744.719196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10745.293366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10750.462265] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10751.112776] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10759.994127] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10762.168236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10768.744359] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10770.755860] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10778.601036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10778.623617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10778.657914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10778.660328] Lustre: Skipped 1 previous similar message [10781.173596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10781.194396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10784.423501] Lustre: server umount lustre-OST0000 complete [10791.393553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10791.396406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10791.412302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10791.425085] Lustre: Skipped 1 previous similar message [10791.444390] Lustre: Skipped 3 previous similar messages [10801.637037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10801.640048] Lustre: Skipped 2 previous similar messages [10805.216833] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10805.448128] Lustre: server umount lustre-MDT0000 complete [10806.759896] LustreError: 325355:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10806.822865] LustreError: 325355:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10811.441123] LustreError: 324539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851889 with bad export cookie 5241368891586480960 [10811.451839] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10811.460945] LustreError: 324539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10812.104329] Lustre: server umount lustre-MDT0001 complete [10831.992539] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [10837.959652] Key type lgssc unregistered [10838.407202] LNet: 328035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10838.419995] LNetError: 328035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10839.467614] LNet: Removed LNI 192.168.202.154@tcp [10840.158933] Key type .llcrypt unregistered [10840.160642] Key type ._llcrypt unregistered [10869.040514] Key type ._llcrypt registered [10869.043659] Key type .llcrypt registered [10869.171241] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [10870.355582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10870.536515] alg: No test for adler32 (adler32-zlib) [10871.878931] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [10872.278839] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [10873.999229] Key type lgssc registered [10875.092407] Lustre: Echo OBD driver; http://www.lustre.org/ [10884.025912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10884.069359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10885.659959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10891.385474] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10895.841283] LustreError: 329455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.862322] LustreError: 329455:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10900.912257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10900.962167] LustreError: 329454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10901.413774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10906.822696] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10915.356955] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10921.630677] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10925.680909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10925.687906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10925.695289] Lustre: Skipped 1 previous similar message [10925.712418] Lustre: Skipped 1 previous similar message [10926.756147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10926.765566] Lustre: Skipped 1 previous similar message [10932.196435] LustreError: 329455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.203983] LustreError: 329455:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10934.022990] Lustre: server umount lustre-MDT0000 complete [10937.316781] LustreError: 329454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.640761] LustreError: 329434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852016 with bad export cookie 4511729939263886620 [10938.656692] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10938.961823] Lustre: server umount lustre-MDT0001 complete [10955.317334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.851198] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10955.935574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10956.452361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10962.534419] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10972.951686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10973.111762] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10973.173501] Lustre: 332042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10973.706492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10979.693252] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [10988.748490] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10995.607734] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11005.812768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11006.064728] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11006.076721] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11006.098806] LustreError: 333926:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [11006.110498] LustreError: 333926:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [11006.123126] LustreError: 333926:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [11006.194785] Lustre: server umount lustre-OST0000 complete [11006.198446] LustreError: 333926:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [11018.359446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11018.654873] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11018.979731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11025.599241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [11027.697400] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11036.741954] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11045.105720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11045.212400] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11045.340878] Lustre: lustre-OST0001: new disk, initializing [11045.347221] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11045.355679] Lustre: 335469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11045.505046] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11053.503365] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11053.764134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11053.778982] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11053.900199] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11062.471547] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11069.408871] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11069.412027] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11069.422806] LustreError: Skipped 1 previous similar message [11069.426501] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11069.444558] Lustre: Skipped 1 previous similar message [11069.465220] Lustre: Skipped 1 previous similar message [11072.511507] Lustre: server umount lustre-OST0001 complete [11074.539762] LustreError: 334392:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11074.552638] LustreError: 334392:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11079.659168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11079.701726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11083.967115] Lustre: server umount lustre-OST0000 complete [11089.891066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11089.923814] Lustre: Skipped 1 previous similar message [11089.931789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11089.939898] Lustre: Skipped 2 previous similar messages [11093.931757] Lustre: server umount lustre-MDT0000 complete [11095.010354] LustreError: 332050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11095.047174] LustreError: 332050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11099.419093] LustreError: 332035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852177 with bad export cookie 4511729939263887180 [11099.430739] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11099.445757] LustreError: 332035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11099.854524] Lustre: server umount lustre-MDT0001 complete [11113.937161] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11116.061690] Key type lgssc unregistered [11116.328814] LNet: 337430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11116.331897] LNetError: 337430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11116.340876] LNet: Removed LNI 192.168.202.154@tcp [11117.130660] Key type .llcrypt unregistered [11117.139041] Key type ._llcrypt unregistered [11205.496509] Key type ._llcrypt registered [11205.498511] Key type .llcrypt registered [11205.576848] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [11222.546515] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11224.539669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11224.585812] alg: No test for adler32 (adler32-zlib) [11225.757296] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [11226.091819] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11227.871246] Key type lgssc registered [11228.830319] Lustre: Echo OBD driver; http://www.lustre.org/ [11280.952499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11280.996438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11282.284405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11282.308229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11282.400531] Lustre: lustre-MDT0000: new disk, initializing [11282.506292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11282.521879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11288.402826] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11300.703107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.880493] Lustre: 342370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11300.948173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11300.951430] Lustre: Skipped 1 previous similar message [11301.037629] Lustre: lustre-MDT0001: new disk, initializing [11301.188258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11301.230905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11301.238799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11305.188277] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11317.959497] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11325.669976] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11332.487309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11332.913859] Lustre: lustre-OST0000: new disk, initializing [11332.917754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11332.935392] Lustre: 344341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11333.049848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11338.850296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11338.874246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11339.008870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11339.784510] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11350.840095] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11356.495597] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11356.796954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11362.714609] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11362.987474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11369.954674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11369.955023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11369.959898] LustreError: Skipped 1 previous similar message [11369.960967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11369.970983] Lustre: Skipped 1 previous similar message [11371.665321] Lustre: server umount lustre-OST0000 complete [11378.153260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11378.168592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11378.201454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11378.217358] Lustre: Skipped 1 previous similar message [11380.193610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11380.195135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11380.242602] Lustre: Skipped 1 previous similar message [11382.054450] Lustre: server umount lustre-MDT0000 complete [11386.167618] LustreError: 342362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852463 with bad export cookie 10602758040634650386 [11386.169907] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11386.175497] LustreError: 342362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11386.657888] Lustre: server umount lustre-MDT0001 complete [11397.309858] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:41:13 (1787852473) [11406.383267] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:41:22 (1787852482) [11415.356308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.829142] LustreError: 347097:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11415.860521] LustreError: 347097:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11415.917414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11420.853482] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11421.166864] LustreError: 347098:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.316463] LustreError: 347097:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.382888] LustreError: 347098:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11431.019684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.540158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11435.760211] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11447.446549] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11456.021815] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11463.335722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11463.924061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11470.528684] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11480.249762] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11495.666856] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11498.313325] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11500.130796] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11502.735488] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11505.361427] Lustre: Failing over lustre-MDT0000 [11505.603809] Lustre: server umount lustre-MDT0000 complete [11506.154298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11506.163767] LustreError: 347097:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.168066] Lustre: Skipped 1 previous similar message [11506.219512] LustreError: 347097:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11516.386829] LustreError: 347094:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11516.410028] LustreError: 347094:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [11521.439656] Lustre: 340592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852583/real 1787852583] req@0000000057b01f58 x1874699016548096/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787852599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11521.464390] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11526.581207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11531.750029] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x932490431b71f697 [11531.764623] Lustre: MGC192.168.202.154@tcp: Connection restored to 0@lo (at 0@lo) [11532.137091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11532.241920] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11534.085560] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11537.397813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11537.461687] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11537.536768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11537.724684] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11550.317952] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11552.676383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11557.227095] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11560.301413] Lustre: Failing over lustre-MDT0000 [11560.669387] Lustre: server umount lustre-MDT0000 complete [11562.977282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11562.979526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11562.985906] LustreError: 347091:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11562.985917] LustreError: 347091:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [11563.012508] Lustre: Skipped 3 previous similar messages [11578.831114] Lustre: 340594:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852640/real 1787852640] req@0000000055740fd9 x1874699016571392/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787852656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11578.845833] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11582.375668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11589.657945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11589.732667] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11590.409902] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11594.735057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11594.744716] Lustre: Skipped 2 previous similar messages [11594.897752] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11594.979083] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11595.051546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11605.676586] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11608.091656] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11620.321589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11620.328477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11620.338478] Lustre: Skipped 1 previous similar message [11620.353485] Lustre: Skipped 2 previous similar messages [11622.893134] Lustre: server umount lustre-MDT0000 complete [11625.445985] LustreError: 347097:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11625.456768] LustreError: 347097:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 24 previous similar messages [11627.191262] LustreError: 347079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852705 with bad export cookie 10602758040634653095 [11627.194659] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11627.214066] LustreError: 347079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11627.915859] Lustre: server umount lustre-MDT0001 complete [11642.665196] Lustre: server umount lustre-OST0000 complete [11661.719796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11662.204217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11669.511788] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11680.341060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11684.489571] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11692.794802] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11699.732733] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11706.915500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11715.092829] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11726.914742] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11731.772185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11739.980792] Lustre: server umount lustre-OST0000 complete [11747.299921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11747.307677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11747.345078] Lustre: Skipped 2 previous similar messages [11747.348250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11747.360779] Lustre: Skipped 1 previous similar message [11754.767936] Lustre: server umount lustre-MDT0000 complete [11756.447136] Lustre: 340592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852818/real 1787852818] req@000000004dfef338 x1874699016626944/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787852834 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11756.482367] LustreError: 354539:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11756.499286] LustreError: 354539:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11758.560914] Lustre: 340593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852820/real 1787852820] req@00000000c56afe26 x1874699016627712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787852836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11758.855890] LustreError: 353457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852836 with bad export cookie 10602758040634653844 [11758.866644] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11758.876797] LustreError: 353457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11759.339448] Lustre: server umount lustre-MDT0001 complete [11772.013289] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [11776.970040] Key type lgssc unregistered [11777.487431] LNet: 357012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11777.491503] LNetError: 357012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11778.538601] LNet: Removed LNI 192.168.202.154@tcp [11780.123456] Key type .llcrypt unregistered [11780.127496] Key type ._llcrypt unregistered [11812.746795] Key type ._llcrypt registered [11812.748042] Key type .llcrypt registered [11812.860869] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [11814.467951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11814.530489] alg: No test for adler32 (adler32-zlib) [11815.530611] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [11815.665443] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [11817.312303] Key type lgssc registered [11818.501194] Lustre: Echo OBD driver; http://www.lustre.org/ [11826.491422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11826.512231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11828.120478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11833.571966] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11837.416848] LustreError: 358530:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11837.454175] LustreError: 358530:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11842.531887] LustreError: 358531:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11844.203969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11844.990713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11851.261759] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11860.294475] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11866.964443] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11874.940640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11875.263673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11881.514114] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11889.170932] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11897.831001] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11911.649863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11911.658827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11911.687216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11913.189899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11913.192401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11913.206398] Lustre: Skipped 1 previous similar message [11913.221811] Lustre: Skipped 1 previous similar message [11917.136623] Lustre: server umount lustre-MDT0000 complete [11918.328770] LustreError: 358527:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11918.353446] LustreError: 358527:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11921.921377] LustreError: 358510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852999 with bad export cookie 6638454882419623320 [11921.927479] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11921.933396] LustreError: 358510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11922.172715] Lustre: server umount lustre-MDT0001 complete [11936.469404] Lustre: server umount lustre-OST0000 complete [11954.012112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11954.510375] LustreError: 362266:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11954.674733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11959.779919] LustreError: 362267:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.843406] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11970.030457] LustreError: 362266:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11970.042789] LustreError: 362266:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11973.869056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11974.234033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11979.343554] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [11989.250823] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11995.901516] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12004.767594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12005.189579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12013.163654] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12023.128616] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12025.665286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [12045.792452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12045.805126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12045.818089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12045.822776] Lustre: Skipped 1 previous similar message [12049.872841] Lustre: server umount lustre-MDT0000 complete [12051.446638] LustreError: 362267:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12051.481473] LustreError: 362267:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12054.675444] LustreError: 362247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853132 with bad export cookie 6638454882419624440 [12054.678607] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12054.693692] LustreError: 362247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12055.205652] Lustre: server umount lustre-MDT0001 complete [12070.093368] Lustre: server umount lustre-OST0000 complete [12088.839290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12089.495891] LustreError: 366180:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.593265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12094.800408] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12105.464606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12111.322570] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12120.028694] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12127.061748] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12135.188282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12135.741051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12135.750531] Lustre: Skipped 1 previous similar message [12136.760887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [12143.686273] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12154.286271] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12161.504492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12161.514887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12161.527860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12161.527870] Lustre: Skipped 3 previous similar messages [12161.555342] Lustre: Skipped 3 previous similar messages [12166.631251] Lustre: server umount lustre-OST0000 complete [12171.749987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12171.758648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12171.771286] Lustre: Skipped 1 previous similar message [12171.802884] Lustre: Skipped 2 previous similar messages [12177.245939] Lustre: server umount lustre-MDT0000 complete [12181.849393] LustreError: 366161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853259 with bad export cookie 6638454882419625462 [12181.875102] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12182.363747] Lustre: server umount lustre-MDT0001 complete [12194.724897] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12198.065717] Key type lgssc unregistered [12198.384511] LNet: 369716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12198.397254] LNetError: 369716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12198.413947] LNet: Removed LNI 192.168.202.154@tcp [12199.326457] Key type .llcrypt unregistered [12199.329723] Key type ._llcrypt unregistered [12229.867547] Key type ._llcrypt registered [12229.872150] Key type .llcrypt registered [12230.020615] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12231.709390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12231.740957] alg: No test for adler32 (adler32-zlib) [12233.026273] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [12233.548837] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12235.231731] Key type lgssc registered [12236.968721] Lustre: Echo OBD driver; http://www.lustre.org/ [12246.850575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12246.862270] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12246.867960] LustreError: 371159:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12246.887338] LustreError: 371159:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12246.899755] LustreError: 371159:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12246.915406] LustreError: 371159:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12246.922239] LustreError: 371159:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12256.546942] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12256.558309] LustreError: 371556:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12256.579705] LustreError: 371556:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12256.603199] LustreError: 371556:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12256.614737] LustreError: 371556:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12256.626329] LustreError: 371556:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12280.954025] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12284.342887] Key type lgssc unregistered [12284.629390] LNet: 372565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12284.640637] LNetError: 372565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12284.661448] LNet: Removed LNI 192.168.202.154@tcp [12285.544153] Key type .llcrypt unregistered [12285.546721] Key type ._llcrypt unregistered [12308.552657] Key type ._llcrypt registered [12308.554440] Key type .llcrypt registered [12308.772675] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [12329.446455] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12331.219339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12331.483769] alg: No test for adler32 (adler32-zlib) [12332.819443] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [12333.271635] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12335.031373] Key type lgssc registered [12337.411050] Lustre: Echo OBD driver; http://www.lustre.org/ [12381.185012] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12381.284456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12382.801715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12382.842317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12382.938875] Lustre: lustre-MDT0000: new disk, initializing [12383.097593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12383.120517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12389.931436] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12404.041594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12404.305254] Lustre: 375724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12404.435871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12404.439219] Lustre: Skipped 1 previous similar message [12404.599935] Lustre: lustre-MDT0001: new disk, initializing [12404.788377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12404.952164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12404.964023] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12411.297287] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12423.387866] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12429.662728] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12438.581504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12439.001762] Lustre: lustre-OST0000: new disk, initializing [12439.008640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12439.015753] Lustre: 377694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12439.133264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12445.980223] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12448.918125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12448.922327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12449.078814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12456.384764] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12462.520311] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12462.859368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12469.692434] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12470.011181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12474.849339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12474.862631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12474.865332] LustreError: Skipped 1 previous similar message [12474.867246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12474.917147] Lustre: Skipped 1 previous similar message [12478.637323] Lustre: server umount lustre-OST0000 complete [12485.088996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12485.092626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12485.102596] Lustre: Skipped 1 previous similar message [12485.108296] Lustre: Skipped 1 previous similar message [12488.906409] Lustre: server umount lustre-MDT0000 complete [12493.672228] LustreError: 377693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853571 with bad export cookie 4247723712044997775 [12493.678384] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12493.688090] LustreError: 377693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12494.212604] Lustre: server umount lustre-MDT0001 complete [12505.303230] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12507.225349] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:59:43 (1787853583) [12515.421528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12515.928631] LustreError: 380155:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.954434] LustreError: 380155:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12516.011170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12521.441021] LustreError: 380156:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12522.026326] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12525.540485] LustreError: 380155:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12530.660756] LustreError: 380156:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.681983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12532.055620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12537.446351] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12546.224476] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12553.965565] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12562.310721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12562.844326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12570.106660] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12580.108194] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12588.402296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12588.689285] Lustre: lustre-OST0001: new disk, initializing [12588.696700] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12588.703238] Lustre: 383007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12588.821186] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12594.297930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12594.311525] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12594.490317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12596.756727] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12608.475841] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12615.153724] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12615.154131] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12615.163612] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12615.163620] Lustre: Skipped 1 previous similar message [12619.171272] Lustre: server umount lustre-OST0001 complete [12620.269978] LustreError: 381983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.290700] LustreError: 381983:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12630.517853] LustreError: 381983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12630.536361] LustreError: 381983:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12638.689019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12638.706492] LustreError: Skipped 1 previous similar message [12638.710290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12638.732031] Lustre: Skipped 1 previous similar message [12638.739041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12638.744791] Lustre: Skipped 2 previous similar messages [12645.872981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12645.885600] Lustre: Skipped 3 previous similar messages [12651.003119] LustreError: 381982:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12651.024399] LustreError: 381982:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [12651.999353] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12652.320668] Lustre: server umount lustre-OST0000 complete [12659.169926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12659.173731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12659.185905] Lustre: Skipped 2 previous similar messages [12659.199398] Lustre: Skipped 3 previous similar messages [12663.283618] Lustre: server umount lustre-MDT0000 complete [12667.309507] LustreError: 380891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853745 with bad export cookie 4247723712044998657 [12667.317025] LustreError: 380891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12667.325216] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12673.938617] Lustre: server umount lustre-MDT0001 complete [12686.375937] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [12690.016280] Key type lgssc unregistered [12690.375911] LNet: 385002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12690.383260] LNetError: 385002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12690.407906] LNet: Removed LNI 192.168.202.154@tcp [12691.606744] Key type .llcrypt unregistered [12691.608559] Key type ._llcrypt unregistered [12747.100444] Key type ._llcrypt registered [12747.103915] Key type .llcrypt registered [12747.247664] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [12748.920154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12749.035134] alg: No test for adler32 (adler32-zlib) [12750.251272] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [12750.532608] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [12752.175192] Key type lgssc registered [12753.150425] Lustre: Echo OBD driver; http://www.lustre.org/ [12760.464646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12760.476335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12761.872471] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12761.904324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12762.024040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12767.589752] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12777.519843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12777.662122] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12777.719418] Lustre: 387802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12778.027787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12782.388360] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12791.736563] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12799.119959] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12803.554870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12803.562234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12803.576022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12806.624712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12806.628207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12806.666463] Lustre: Skipped 1 previous similar message [12811.744717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12816.351135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12816.511910] Lustre: server umount lustre-MDT0000 complete [12820.904325] LustreError: 387797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853898 with bad export cookie 14366223161985253355 [12820.910973] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12820.924492] LustreError: 387797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12821.188332] Lustre: server umount lustre-MDT0001 complete [12832.306545] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:05:08 (1787853908) [12840.791652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.323376] LustreError: 390317:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12841.358829] LustreError: 390317:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12841.466641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12846.578366] LustreError: 390318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12846.598164] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12851.688164] LustreError: 390317:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12856.802430] LustreError: 390318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12856.907443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12857.271300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12862.252357] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12870.303696] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12877.079527] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12884.889449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12885.113762] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12885.385239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12891.749775] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12898.873589] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12906.575708] Lustre: Permanently deactivating lustre-OST0000 [12906.590506] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12912.676708] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12914.930984] LustreError: 393253:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12914.941484] LustreError: 393253:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12914.946467] LustreError: 393253:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12918.358707] LustreError: 393302:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12918.372617] LustreError: 393302:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12926.544289] Lustre: server umount lustre-OST0000 complete [12929.558494] LustreError: 393552:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12929.563924] LustreError: 393552:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12934.113369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12934.120509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12934.149591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12934.178179] Lustre: Skipped 1 previous similar message [12936.161605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12941.283971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12941.302905] Lustre: Skipped 3 previous similar messages [12941.459926] Lustre: server umount lustre-MDT0000 complete [12946.116736] LustreError: 390300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854023 with bad export cookie 14366223161985253936 [12946.118861] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12946.132585] LustreError: 390300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12955.757531] Lustre: server umount lustre-MDT0001 complete [12966.448031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12972.334601] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [12975.613625] LustreError: 394715:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12982.478950] Lustre: 395053:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.154@tcp [12985.138251] Lustre: server umount MGS complete [12997.781529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12998.234263] LustreError: 395777:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12998.272719] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12998.285221] Lustre: Skipped 1 previous similar message [12998.398313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13003.309650] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13008.868674] LustreError: 395778:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13008.893423] LustreError: 395778:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13013.243100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13013.722944] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13019.690778] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13028.815954] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13034.873189] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13037.187297] Lustre: Permanently reactivating lustre-OST0000 [13037.200924] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [13037.204874] Lustre: Skipped 2 previous similar messages [13065.695814] LustreError: 397557:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [13073.586377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13073.943407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13073.947720] Lustre: Skipped 1 previous similar message [13080.527918] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13087.915167] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13095.571228] Lustre: server umount lustre-OST0000 complete [13101.025462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13101.033578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13101.043702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13112.287250] Lustre: 387127:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854174/real 1787854174] req@00000000fc2bb7f3 x1874700615647360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787854190 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13112.320573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13112.330051] Lustre: Skipped 1 previous similar message [13112.799797] Lustre: 387125:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854174/real 1787854174] req@0000000089034897 x1874700615646592/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787854190 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13114.336585] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13114.519632] Lustre: server umount lustre-MDT0000 complete [13117.410957] LustreError: 395772:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13117.423376] LustreError: 395772:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13117.985442] Lustre: 387125:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854179/real 1787854179] req@0000000082b9c332 x1874700615649152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787854195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13118.489050] LustreError: 395759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854196 with bad export cookie 14366223161985254986 [13118.495625] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13118.500061] LustreError: 395759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13118.868417] Lustre: server umount lustre-MDT0001 complete [13130.205680] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13133.136294] Key type lgssc unregistered [13133.437191] LNet: 400146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13133.450717] LNetError: 400146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13133.482959] LNet: Removed LNI 192.168.202.154@tcp [13134.357956] Key type .llcrypt unregistered [13134.361698] Key type ._llcrypt unregistered [13153.853863] Key type ._llcrypt registered [13153.856331] Key type .llcrypt registered [13154.012399] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [13172.249872] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13173.380551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13173.497333] alg: No test for adler32 (adler32-zlib) [13174.652354] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [13175.021350] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13176.767521] Key type lgssc registered [13177.822205] Lustre: Echo OBD driver; http://www.lustre.org/ [13212.937600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13212.971184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13214.288746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13214.333749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13214.417948] Lustre: lustre-MDT0000: new disk, initializing [13214.476783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13214.497795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13219.731915] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13230.928510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13231.092849] Lustre: 403304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13231.163931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13231.172878] Lustre: Skipped 1 previous similar message [13231.262714] Lustre: lustre-MDT0001: new disk, initializing [13231.395565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13231.454917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13231.471501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13236.242933] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13246.985820] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13252.951535] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13260.001902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13260.332695] Lustre: lustre-OST0000: new disk, initializing [13260.338843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13260.344606] Lustre: 405273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13260.469771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13266.118816] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13268.607239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13268.631677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13268.740499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13275.964632] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13280.346589] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13280.595557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13284.931310] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13285.180799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13289.443168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13289.447070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13289.451253] LustreError: Skipped 1 previous similar message [13289.463176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13289.475046] Lustre: Skipped 1 previous similar message [13293.294105] Lustre: server umount lustre-OST0000 complete [13298.146752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13298.160558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13298.180209] Lustre: Skipped 1 previous similar message [13298.192992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13299.680811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13299.684169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13299.703917] Lustre: Skipped 1 previous similar message [13304.132202] Lustre: server umount lustre-MDT0000 complete [13308.112352] LustreError: 405272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854385 with bad export cookie 11623767763588231536 [13308.124684] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13308.500174] Lustre: server umount lustre-MDT0001 complete [13317.377227] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13318.711965] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:13:15 (1787854395) [13326.368725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13326.665706] LustreError: 407732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13326.679030] LustreError: 407732:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13326.719296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13331.561215] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13331.937329] LustreError: 407733:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13337.068843] LustreError: 407732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13340.703408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13341.206889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13345.851865] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13353.577700] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13359.425786] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13366.154204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13366.557914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13373.554788] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13380.243336] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13382.032596] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13384.723383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13396.452071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13396.468359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13396.470845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13396.484426] Lustre: Skipped 1 previous similar message [13397.550953] Lustre: server umount lustre-OST0000 complete [13401.572165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13401.583072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13401.598586] Lustre: Skipped 2 previous similar messages [13401.621733] Lustre: Skipped 1 previous similar message [13411.809827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13411.813522] Lustre: Skipped 2 previous similar messages [13415.391405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13415.551434] Lustre: server umount lustre-MDT0000 complete [13416.932071] LustreError: 407732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13416.947921] LustreError: 407732:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13419.187770] LustreError: 407713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854497 with bad export cookie 11623767763588232418 [13419.197767] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13419.200235] LustreError: 407713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13419.566520] Lustre: server umount lustre-MDT0001 complete [13430.554749] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13433.386668] Key type lgssc unregistered [13433.695579] LNet: 411462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13433.710850] LNetError: 411462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13433.750470] LNet: Removed LNI 192.168.202.154@tcp [13434.725600] Key type .llcrypt unregistered [13434.728066] Key type ._llcrypt unregistered [13457.973857] Key type ._llcrypt registered [13457.976978] Key type .llcrypt registered [13458.094889] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13458.357510] Key type .llcrypt unregistered [13458.359783] Key type ._llcrypt unregistered [13474.509455] Key type ._llcrypt registered [13474.511254] Key type .llcrypt registered [13474.603228] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13475.666136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13476.040638] alg: No test for adler32 (adler32-zlib) [13477.116748] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [13477.490633] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13479.167189] Key type lgssc registered [13480.572200] Lustre: Echo OBD driver; http://www.lustre.org/ [13487.708605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13487.728585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13489.129302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13493.859412] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13498.338127] LustreError: 413678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13498.371531] LustreError: 413678:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13502.629406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13502.999225] LustreError: 413679:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13503.030302] LustreError: 413679:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13503.048827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13508.072747] LustreError: 413678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13508.088093] LustreError: 413678:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13510.352482] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13513.196696] LustreError: 414422:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13513.218929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13513.219720] LustreError: 414422:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13518.348634] LustreError: 413679:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13518.370455] LustreError: 413679:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13518.954915] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13526.455414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13526.759154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13531.291156] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13541.923944] Lustre: server umount lustre-OST0000 complete [13545.870923] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13545.884546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13545.904789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13547.490946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13547.494565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13547.525555] Lustre: Skipped 1 previous similar message [13552.609390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13555.754371] Lustre: server umount lustre-MDT0000 complete [13557.729102] LustreError: 413678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13557.748710] LustreError: 413678:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13558.751348] Lustre: 412992:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854620/real 1787854620] req@000000009b47b5b0 x1874701377890688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787854636 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13558.769499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13559.528284] LustreError: 413658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854637 with bad export cookie 3367737324636928754 [13559.545489] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13559.822849] Lustre: server umount lustre-MDT0001 complete [13570.169296] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13573.347604] Key type lgssc unregistered [13573.664624] LNet: 416828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13573.670282] LNetError: 416828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13573.690922] LNet: Removed LNI 192.168.202.154@tcp [13574.559981] Key type .llcrypt unregistered [13574.562675] Key type ._llcrypt unregistered [13601.815268] Key type ._llcrypt registered [13601.817373] Key type .llcrypt registered [13601.961493] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13602.873165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13602.913664] alg: No test for adler32 (adler32-zlib) [13604.090789] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [13604.419177] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13606.135450] Key type lgssc registered [13607.402561] Lustre: Echo OBD driver; http://www.lustre.org/ [13615.214309] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13615.227709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13630.815725] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13635.937939] LustreError: 418219:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [13662.687225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13670.351184] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13679.073876] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13685.693458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13686.118282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13686.230353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13690.598155] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13691.367088] LustreError: 418245:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13691.389653] LustreError: 418245:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13696.480797] LustreError: 418244:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13697.533553] LustreError: 419259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13697.551427] LustreError: 419259:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13699.509770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13700.039694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13704.741745] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13713.658824] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13719.318332] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13729.822785] Lustre: server umount lustre-OST0000 complete [13733.989941] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13734.002848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13734.025120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13735.905691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13735.913536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13741.028162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13741.039271] Lustre: Skipped 2 previous similar messages [13744.186240] Lustre: server umount lustre-MDT0000 complete [13746.144222] LustreError: 419263:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13746.168950] LustreError: 419263:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13747.117286] Lustre: 417556:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854808/real 1787854808] req@0000000070521d24 x1874701511064832/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787854824 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13747.144330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13747.931227] LustreError: 418225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854825 with bad export cookie 6306986623218290560 [13747.942269] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13747.944457] LustreError: 418225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13748.453795] Lustre: server umount lustre-MDT0001 complete [13760.416402] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13762.902558] Key type lgssc unregistered [13763.147424] LNet: 421792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13763.153796] LNetError: 421792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13763.163281] LNet: Removed LNI 192.168.202.154@tcp [13763.832606] Key type .llcrypt unregistered [13763.834624] Key type ._llcrypt unregistered [13787.123328] Key type ._llcrypt registered [13787.128114] Key type .llcrypt registered [13787.225905] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13788.206847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13788.321382] alg: No test for adler32 (adler32-zlib) [13789.400318] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [13789.711712] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13791.391188] Key type lgssc registered [13792.452391] Lustre: Echo OBD driver; http://www.lustre.org/ [13798.863272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13798.872207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13800.321609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13804.555830] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13810.657629] LustreError: 423209:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13810.688447] LustreError: 423209:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13812.743028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13813.188946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13817.984554] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13827.003887] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13833.969724] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13842.201439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13842.721845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13845.817117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13849.695664] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13857.770956] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13863.401298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13863.404777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13863.415885] Lustre: Skipped 2 previous similar messages [13863.432043] Lustre: Skipped 2 previous similar messages [13865.751397] LustreError: 423206:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13865.923706] Lustre: server umount lustre-MDT0000 complete [13868.517719] LustreError: 423205:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13868.548946] LustreError: 423205:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13868.569370] LustreError: 423189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.54@tcp arrived at 1787854946 with bad export cookie 15688362914051000795 [13868.593602] LustreError: 423189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13870.683089] LustreError: 423188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854948 with bad export cookie 15688362914050999899 [13870.703184] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13873.632348] LustreError: 424211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13873.637819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13873.660234] LustreError: 424211:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13873.669956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13873.674082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13875.683326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13877.135871] Lustre: server umount lustre-OST0000 complete [13880.806803] LustreError: 424211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13880.824380] LustreError: 424211:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13885.797265] Lustre: server umount lustre-MDT0001 complete [13899.799672] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [13902.967188] Key type lgssc unregistered [13903.261340] LNet: 426899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13903.272280] LNetError: 426899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13903.296304] LNet: Removed LNI 192.168.202.154@tcp [13904.074920] Key type .llcrypt unregistered [13904.078214] Key type ._llcrypt unregistered [13929.243694] Key type ._llcrypt registered [13929.245778] Key type .llcrypt registered [13929.347850] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [13930.837490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13930.888307] alg: No test for adler32 (adler32-zlib) [13931.829876] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [13932.037896] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [13933.767530] Key type lgssc registered [13934.960280] Lustre: Echo OBD driver; http://www.lustre.org/ [13943.709510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13943.735907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13945.222679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13950.415870] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13955.566543] LustreError: 428319:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13955.601255] LustreError: 428319:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13960.681685] LustreError: 428318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13964.178324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13964.786107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13972.344963] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [13981.697622] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13989.292472] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13996.689623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13997.039657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14003.225635] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14011.148603] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14015.494260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14020.587758] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14020.599383] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14020.602246] Lustre: Skipped 1 previous similar message [14022.965390] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [14025.704286] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14025.717812] Lustre: Skipped 1 previous similar message [14025.942734] Lustre: server umount lustre-MDT0001 complete [14028.032366] LustreError: 428314:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14030.821583] LustreError: 428318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14030.835012] LustreError: 428318:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14035.941626] LustreError: 429047:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14035.946586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14035.953061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14035.977312] Lustre: Skipped 1 previous similar message [14035.982483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14038.360626] Lustre: server umount lustre-OST0000 complete [14042.307813] Lustre: server umount lustre-MDT0000 complete [14056.986806] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [14059.796274] Key type lgssc unregistered [14060.184808] LNet: 432013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14060.190739] LNetError: 432013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14060.237075] LNet: Removed LNI 192.168.202.154@tcp [14061.231704] Key type .llcrypt unregistered [14061.235624] Key type ._llcrypt unregistered [14088.598680] Key type ._llcrypt registered [14088.600830] Key type .llcrypt registered [14088.698156] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [14104.115498] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [14105.658873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14105.757411] alg: No test for adler32 (adler32-zlib) [14106.974572] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [14107.250327] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [14108.903182] Key type lgssc registered [14110.416536] Lustre: Echo OBD driver; http://www.lustre.org/ [14151.579542] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [14169.079131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14169.129910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14170.516575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14170.570155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14170.692465] Lustre: lustre-MDT0000: new disk, initializing [14170.816398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14170.838928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14175.699491] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14189.952895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14190.227420] Lustre: lustre-OST0000: new disk, initializing [14190.232425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14190.236376] Lustre: Skipped 1 previous similar message [14190.240908] Lustre: 437051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14190.300375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14196.634985] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14198.879258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14198.884598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14198.909829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14205.294725] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14221.196612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14221.360168] Lustre: 436033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14221.489479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14221.619179] Lustre: lustre-MDT0001: new disk, initializing [14221.826962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14221.880687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14221.906492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14227.163207] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14237.083765] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14237.328343] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14241.706518] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14242.003446] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14247.392411] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14247.403667] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14247.409859] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14247.412568] Lustre: Skipped 1 previous similar message [14247.422688] Lustre: Skipped 1 previous similar message [14248.730304] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [14252.513986] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14253.501501] Lustre: server umount lustre-MDT0001 complete [14262.765354] LustreError: 436039:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14262.795051] LustreError: 436039:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [14267.875858] LustreError: 436039:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14267.890946] LustreError: 436039:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14272.995618] LustreError: 436043:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14273.014726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14273.022833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14273.026354] Lustre: Skipped 1 previous similar message [14277.632662] Lustre: server umount lustre-OST0000 complete [14278.113708] LustreError: 436044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14287.740753] Lustre: server umount lustre-MDT0000 complete [14303.577470] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [14306.439371] Key type lgssc unregistered [14306.744051] LNet: 440802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14306.758570] LNetError: 440802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14306.773555] LNet: Removed LNI 192.168.202.154@tcp [14307.826339] Key type .llcrypt unregistered [14307.827916] Key type ._llcrypt unregistered [14346.562996] Key type ._llcrypt registered [14346.566303] Key type .llcrypt registered [14346.666654] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [14347.032647] Key type .llcrypt unregistered [14347.040079] Key type ._llcrypt unregistered [14361.579398] Key type ._llcrypt registered [14361.581571] Key type .llcrypt registered [14361.666656] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [14362.416732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14362.442376] alg: No test for adler32 (adler32-zlib) [14363.872865] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [14364.138626] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [14365.895348] Key type lgssc registered [14367.389113] Lustre: Echo OBD driver; http://www.lustre.org/ [14395.791661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14395.840566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14397.219321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14397.262990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14397.426699] Lustre: lustre-MDT0000: new disk, initializing [14397.534932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14397.589268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14403.252430] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14416.540527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14416.743587] Lustre: 444184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14416.863715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14416.867028] Lustre: Skipped 1 previous similar message [14416.923966] Lustre: lustre-MDT0001: new disk, initializing [14417.012462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14417.063628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14417.072659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14421.758888] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14431.441313] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14437.700421] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14444.505719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14444.745924] Lustre: lustre-OST0000: new disk, initializing [14444.749478] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14444.754946] Lustre: 446151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14444.870507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14450.623956] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14452.300093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14452.316391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14452.427117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14459.620603] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14468.075612] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14468.079423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14468.080563] Lustre: Skipped 1 previous similar message [14468.080948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14468.098981] Lustre: Skipped 2 previous similar messages [14473.250491] Lustre: server umount lustre-MDT0000 complete [14477.536976] LustreError: 446150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855555 with bad export cookie 104882349162347269 [14477.541417] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14477.551352] LustreError: 446150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14477.939092] Lustre: server umount lustre-MDT0001 complete [14492.237349] Lustre: server umount lustre-OST0000 complete [14504.270700] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:33:00 (1787855580) [14512.110987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14512.514851] LustreError: 448179:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14512.530680] LustreError: 448179:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14512.582781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14517.739166] LustreError: 448180:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14517.898120] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14522.875579] LustreError: 448179:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14527.990202] LustreError: 448180:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14531.227723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14531.788614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14536.515826] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14543.618342] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14548.787189] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14567.935542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14568.120342] Lustre: Found index 0 for lustre-OST0000, updating log [14568.325127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14574.357659] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [14583.895165] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14584.828166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14595.041057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14595.046268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14595.049747] Lustre: Skipped 2 previous similar messages [14595.064984] Lustre: Skipped 1 previous similar message [14596.129965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14596.136820] Lustre: Skipped 1 previous similar message [14601.392393] Lustre: server umount lustre-OST0000 complete [14617.570335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14617.579707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14617.583917] Lustre: Skipped 1 previous similar message [14622.696384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14622.705391] Lustre: Skipped 1 previous similar message [14623.252605] Lustre: server umount lustre-MDT0000 complete [14623.718452] LustreError: 448175:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14623.758294] LustreError: 448175:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14627.362473] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp [14628.320342] LustreError: 448159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855706 with bad export cookie 104882349162348662 [14628.325360] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14628.332661] LustreError: 448159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14628.928724] Lustre: server umount lustre-MDT0001 complete [14639.817303] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:35:15 (1787855715) [15266.176397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15266.886179] LustreError: 454597:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15266.941887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15271.407055] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15272.424306] LustreError: 454598:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15276.517878] LustreError: 454597:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15281.140622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15281.713540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15286.742955] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15295.236867] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15302.166457] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15318.559686] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp [15323.759785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15324.100867] Lustre: Found index 0 for lustre-OST0000, updating log [15324.451343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15331.434939] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15339.565119] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15342.342414] Lustre: server umount lustre-OST0000 complete [15347.041219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15347.059322] Lustre: Skipped 1 previous similar message [15347.070204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15347.081860] Lustre: Skipped 1 previous similar message [15348.147083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15348.160166] Lustre: Skipped 1 previous similar message [15350.248062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15350.266714] Lustre: Skipped 3 previous similar messages [15352.617829] Lustre: server umount lustre-MDT0000 complete [15353.319316] LustreError: 455580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15353.343567] LustreError: 455580:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15357.595097] LustreError: 454578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856435 with bad export cookie 104882349162349747 [15357.603665] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15358.145264] Lustre: server umount lustre-MDT0001 complete [15378.975337] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp [15389.728722] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [15404.501947] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15452.360928] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:48:48 (1787856528) [15470.761123] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [15473.505917] Key type lgssc unregistered [15473.879906] LNet: 461209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15473.893447] LNetError: 461209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15586.532879] LNet: Removed LNI 192.168.202.154@tcp [15587.896945] Key type .llcrypt unregistered [15587.902303] Key type ._llcrypt unregistered [15606.656313] Key type ._llcrypt registered [15606.658143] Key type .llcrypt registered [15606.808401] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15607.852389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15607.878250] alg: No test for adler32 (adler32-zlib) [15609.150775] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [15609.458896] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [15611.144196] Key type lgssc registered [15612.633698] Lustre: Echo OBD driver; http://www.lustre.org/ [15617.561689] LNet: Added LNI 192.168.202.154@tcp100 [8/256/0/180] [15618.638046] LNet: Added LNI 192.168.202.154@tcp101 [8/256/0/180] [15619.848123] LNet: Added LNI 192.168.202.154@tcp102 [8/256/0/180] [15620.866067] LNet: Added LNI 192.168.202.154@tcp103 [8/256/0/180] [15622.303492] LNet: Added LNI 192.168.202.154@tcp104 [8/256/0/180] [15623.289795] LNet: Added LNI 192.168.202.154@tcp105 [8/256/0/180] [15624.357886] LNet: Added LNI 192.168.202.154@tcp106 [8/256/0/180] [15625.612517] LNet: Added LNI 192.168.202.154@tcp107 [8/256/0/180] [15626.675206] LNet: Added LNI 192.168.202.154@tcp108 [8/256/0/180] [15627.686330] LNet: Added LNI 192.168.202.154@tcp109 [8/256/0/180] [15628.727142] LNet: Added LNI 192.168.202.154@tcp110 [8/256/0/180] [15629.737565] LNet: Added LNI 192.168.202.154@tcp111 [8/256/0/180] [15630.667723] LNet: Added LNI 192.168.202.154@tcp112 [8/256/0/180] [15631.707718] LNet: Added LNI 192.168.202.154@tcp113 [8/256/0/180] [15632.826914] LNet: Added LNI 192.168.202.154@tcp114 [8/256/0/180] [15633.890989] LNet: Added LNI 192.168.202.154@tcp115 [8/256/0/180] [15635.010399] LNet: Added LNI 192.168.202.154@tcp116 [8/256/0/180] [15636.173327] LNet: Added LNI 192.168.202.154@tcp117 [8/256/0/180] [15637.336679] LNet: Added LNI 192.168.202.154@tcp118 [8/256/0/180] [15638.274948] LNet: Added LNI 192.168.202.154@tcp119 [8/256/0/180] [15639.418161] LNet: Added LNI 192.168.202.154@tcp120 [8/256/0/180] [15640.532411] LNet: Added LNI 192.168.202.154@tcp121 [8/256/0/180] [15641.706705] LNet: Added LNI 192.168.202.154@tcp122 [8/256/0/180] [15642.856361] LNet: Added LNI 192.168.202.154@tcp123 [8/256/0/180] [15643.869202] LNet: Added LNI 192.168.202.154@tcp124 [8/256/0/180] [15644.854711] LNet: Added LNI 192.168.202.154@tcp125 [8/256/0/180] [15645.887313] LNet: Added LNI 192.168.202.154@tcp126 [8/256/0/180] [15647.012960] LNet: Added LNI 192.168.202.154@tcp127 [8/256/0/180] [15647.977666] LNet: Added LNI 192.168.202.154@tcp128 [8/256/0/180] [15648.970423] LNet: Added LNI 192.168.202.154@tcp129 [8/256/0/180] [15649.973556] LNet: Added LNI 192.168.202.154@tcp130 [8/256/0/180] [15650.916080] LNet: Added LNI 192.168.202.154@tcp131 [8/256/0/180] [15651.932718] LNet: Added LNI 192.168.202.154@tcp132 [8/256/0/180] [15652.896877] LNet: Added LNI 192.168.202.154@tcp133 [8/256/0/180] [15653.789722] LNet: Added LNI 192.168.202.154@tcp134 [8/256/0/180] [15654.785384] LNet: Added LNI 192.168.202.154@tcp135 [8/256/0/180] [15661.655577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15661.722153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15662.876702] Lustre: 464241:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15663.023887] LNetError: 464264:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15663.064073] LNetError: Protocol error connecting to 192.168.202.154@tcp124 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15663.195896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15663.249860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15663.409673] Lustre: lustre-MDT0000: new disk, initializing [15663.464846] Lustre: 464300:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15663.480509] Lustre: 464300:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15663.504803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15663.521650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15663.780546] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp124 [15664.032873] LNetError: 464272:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15664.046190] LNetError: Protocol error connecting to 192.168.202.154@tcp122 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15666.850666] LNetError: 461696:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15666.870628] LNetError: 461696:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15666.884332] LNetError: Protocol error connecting to 192.168.202.154@tcp124 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15666.901831] LNetError: Skipped 1 previous similar message [15669.235496] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15670.945617] LNetError: 461697:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15670.969846] LNetError: Protocol error connecting to 192.168.202.154@tcp124 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15678.122717] LNetError: 461698:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15678.147573] LNetError: Protocol error connecting to 192.168.202.154@tcp124 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15680.798827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15680.973905] Lustre: 464256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15681.037443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15681.046084] Lustre: Skipped 1 previous similar message [15681.163793] Lustre: lustre-MDT0001: new disk, initializing [15681.208620] Lustre: 465062:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15681.214738] Lustre: 465062:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15681.361694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15681.428861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15681.440824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15686.516745] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15691.326656] Lustre: 465488:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15691.332450] Lustre: 465488:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15694.507992] LNetError: 461699:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15694.521704] LNetError: Protocol error connecting to 192.168.202.154@tcp124 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15697.866995] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15704.386744] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15710.821941] LNetError: 464272:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15710.834788] LNetError: 464272:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15710.844324] LNetError: Protocol error connecting to 192.168.202.154@tcp122 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15710.852342] LNetError: Skipped 1 previous similar message [15711.751538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15712.171569] Lustre: lustre-OST0000: new disk, initializing [15712.173619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15712.178527] Lustre: 466235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15712.326202] Lustre: 466241:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15712.330453] Lustre: 466241:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15712.399599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15719.491459] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15722.599713] Lustre: 466693:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.154@tcp: can't add NID 192.168.202.154@tcp130: rc = -75 [15722.616725] Lustre: 466693:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15722.705169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15722.719795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15722.943340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15723.187554] LNet: 1 local NIs in recovery (showing 1): 192.168.202.154@tcp124 [15730.581408] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15743.457144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15743.459123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15743.468955] LustreError: Skipped 1 previous similar message [15743.470085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15743.508550] Lustre: Skipped 1 previous similar message [15744.970911] Lustre: server umount lustre-OST0000 complete [15749.734198] LNetError: 464270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.154 [15749.743776] LNetError: 464270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15749.753601] LNetError: Protocol error connecting to 192.168.202.154@tcp122 at host 192.168.202.154:988: is it running a compatible version of Lustre? [15749.764260] LNetError: Skipped 2 previous similar messages [15753.187278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15753.204150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15753.236206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15753.250366] Lustre: Skipped 1 previous similar message [15756.808235] Lustre: server umount lustre-MDT0000 complete [15760.430280] LustreError: 464250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856838 with bad export cookie 11080735595617717058 [15760.442475] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15760.751186] Lustre: server umount lustre-MDT0001 complete [15774.119480] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [15778.585701] Key type lgssc unregistered [15779.260219] LNet: 468152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15779.269823] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15779.276507] LNet: Removed LNI 192.168.202.154@tcp [15780.321100] LNet: Removed LNI 192.168.202.154@tcp100 [15780.325708] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15780.339906] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15780.354639] LNet: Removed LNI 192.168.202.154@tcp101 [15781.408716] LNet: Removed LNI 192.168.202.154@tcp102 [15781.420983] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15781.438745] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15781.461174] LNet: Removed LNI 192.168.202.154@tcp103 [15782.495446] LNet: Removed LNI 192.168.202.154@tcp104 [15783.519517] LNet: Removed LNI 192.168.202.154@tcp105 [15783.527376] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15783.546340] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15783.556262] LNet: Removed LNI 192.168.202.154@tcp106 [15784.608574] LNet: Removed LNI 192.168.202.154@tcp107 [15785.631174] LNet: Removed LNI 192.168.202.154@tcp108 [15786.659248] LNet: Removed LNI 192.168.202.154@tcp109 [15787.679310] LNet: Removed LNI 192.168.202.154@tcp110 [15787.681360] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15787.694990] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15787.709080] LNet: Removed LNI 192.168.202.154@tcp111 [15788.830366] LNet: Removed LNI 192.168.202.154@tcp112 [15789.857389] LNet: Removed LNI 192.168.202.154@tcp113 [15790.879918] LNet: Removed LNI 192.168.202.154@tcp114 [15791.903729] LNet: Removed LNI 192.168.202.154@tcp115 [15792.927205] LNet: Removed LNI 192.168.202.154@tcp116 [15793.951388] LNet: Removed LNI 192.168.202.154@tcp117 [15795.039318] LNet: Removed LNI 192.168.202.154@tcp118 [15796.066320] LNet: Removed LNI 192.168.202.154@tcp119 [15796.067764] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15796.083275] LNetError: 468152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15796.093723] LNet: Removed LNI 192.168.202.154@tcp120 [15797.151124] LNet: Removed LNI 192.168.202.154@tcp121 [15798.175156] LNet: Removed LNI 192.168.202.154@tcp122 [15799.200882] LNet: Removed LNI 192.168.202.154@tcp123 [15799.203950] LNet: Removed LNI 192.168.202.154@tcp124 [15800.223233] LNet: Removed LNI 192.168.202.154@tcp125 [15801.247214] LNet: Removed LNI 192.168.202.154@tcp126 [15802.271138] LNet: Removed LNI 192.168.202.154@tcp127 [15803.296184] LNet: Removed LNI 192.168.202.154@tcp128 [15804.383150] LNet: Removed LNI 192.168.202.154@tcp129 [15804.386240] LNet: Removed LNI 192.168.202.154@tcp130 [15804.395431] LNet: Removed LNI 192.168.202.154@tcp131 [15804.405754] LNet: Removed LNI 192.168.202.154@tcp132 [15804.412758] LNet: Removed LNI 192.168.202.154@tcp133 [15804.422864] LNet: Removed LNI 192.168.202.154@tcp134 [15804.436314] LNet: Removed LNI 192.168.202.154@tcp135 [15805.961488] Key type .llcrypt unregistered [15805.963520] Key type ._llcrypt unregistered [15831.881876] Key type ._llcrypt registered [15831.886433] Key type .llcrypt registered [15832.006367] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [15847.392336] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15848.840874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15848.876837] alg: No test for adler32 (adler32-zlib) [15849.791221] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [15849.977579] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [15851.584218] Key type lgssc registered [15852.757930] Lustre: Echo OBD driver; http://www.lustre.org/ [15896.251581] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:56:12 (1787856972) [15916.619967] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [15919.490584] Key type lgssc unregistered [15919.789424] LNet: 472247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15919.807988] LNetError: 472247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15919.822621] LNet: Removed LNI 192.168.202.154@tcp [15920.634411] Key type .llcrypt unregistered [15920.638409] Key type ._llcrypt unregistered [15937.095220] Key type ._llcrypt registered [15937.101276] Key type .llcrypt registered [15937.210913] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [15938.364759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15938.379517] alg: No test for adler32 (adler32-zlib) [15939.374767] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [15939.562834] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [15941.199184] Key type lgssc registered [15942.305658] Lustre: Echo OBD driver; http://www.lustre.org/ [15954.015109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15954.106946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15955.437222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15955.489387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15955.599414] Lustre: lustre-MDT0000: new disk, initializing [15955.714294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15955.746496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15962.502598] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15973.586228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15973.830288] Lustre: 473462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15973.938692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15973.943684] Lustre: Skipped 1 previous similar message [15974.073492] Lustre: lustre-MDT0001: new disk, initializing [15974.237773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15974.336958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15974.354484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15980.947855] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [15991.503518] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15997.527791] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16005.066818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16005.372979] Lustre: lustre-OST0000: new disk, initializing [16005.376710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16005.385724] Lustre: 475430:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16005.497156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16012.725787] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16012.924642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16012.943745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16013.060111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16023.761818] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16027.736817] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing lnet_if_list [16033.195276] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16033.224648] LNet: Added LNI 192.168.202.154@tcp10 [8/256/0/180] [16039.351927] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16039.377407] LNet: Added LNI 192.168.202.154@tcp11 [8/256/0/180] [16043.627190] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16043.655710] LNet: Added LNI 192.168.202.154@tcp12 [8/256/0/180] [16048.642961] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16048.654759] LNet: Added LNI 192.168.202.154@tcp13 [8/256/0/180] [16053.536934] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16053.569897] LNet: Added LNI 192.168.202.154@tcp14 [8/256/0/180] [16058.270713] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16058.293994] LNet: Added LNI 192.168.202.154@tcp15 [8/256/0/180] [16063.442151] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16063.464982] LNet: Added LNI 192.168.202.154@tcp16 [8/256/0/180] [16069.790405] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16069.836403] LNet: Added LNI 192.168.202.154@tcp17 [8/256/0/180] [16075.877119] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16075.909915] LNet: Added LNI 192.168.202.154@tcp18 [8/256/0/180] [16081.992705] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16082.030623] LNet: Added LNI 192.168.202.154@tcp19 [8/256/0/180] [16089.255095] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16089.311434] LNet: Added LNI 192.168.202.154@tcp20 [8/256/0/180] [16096.646737] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16096.665513] LNet: Added LNI 192.168.202.154@tcp21 [8/256/0/180] [16101.648415] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16101.663568] LNet: Added LNI 192.168.202.154@tcp22 [8/256/0/180] [16107.363974] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16107.389463] LNet: Added LNI 192.168.202.154@tcp23 [8/256/0/180] [16113.643788] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16113.668110] LNet: Added LNI 192.168.202.154@tcp24 [8/256/0/180] [16120.353861] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16120.399241] LNet: Added LNI 192.168.202.154@tcp25 [8/256/0/180] [16126.732235] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16126.765843] LNet: Added LNI 192.168.202.154@tcp26 [8/256/0/180] [16133.255028] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16133.328226] LNet: Added LNI 192.168.202.154@tcp27 [8/256/0/180] [16140.519491] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16140.571287] LNet: Added LNI 192.168.202.154@tcp28 [8/256/0/180] [16147.373399] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16147.400846] LNet: Added LNI 192.168.202.154@tcp29 [8/256/0/180] [16153.989896] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16154.011726] LNet: Added LNI 192.168.202.154@tcp30 [8/256/0/180] [16160.572285] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16160.606870] LNet: Added LNI 192.168.202.154@tcp31 [8/256/0/180] [16168.654300] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16168.707391] LNet: Added LNI 192.168.202.154@tcp32 [8/256/0/180] [16176.057885] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16176.091810] LNet: Added LNI 192.168.202.154@tcp33 [8/256/0/180] [16183.555041] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16183.580878] LNet: Added LNI 192.168.202.154@tcp34 [8/256/0/180] [16191.312915] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16191.333359] LNet: Added LNI 192.168.202.154@tcp35 [8/256/0/180] [16198.673727] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16198.717968] LNet: Added LNI 192.168.202.154@tcp36 [8/256/0/180] [16206.158528] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16206.186523] LNet: Added LNI 192.168.202.154@tcp37 [8/256/0/180] [16214.048611] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16214.073238] LNet: Added LNI 192.168.202.154@tcp38 [8/256/0/180] [16222.959812] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16223.031499] LNet: Added LNI 192.168.202.154@tcp39 [8/256/0/180] [16232.152032] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16232.190318] LNet: Added LNI 192.168.202.154@tcp40 [8/256/0/180] [16239.595952] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16239.624432] LNet: Added LNI 192.168.202.154@tcp41 [8/256/0/180] [16248.508777] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16248.541723] LNet: Added LNI 192.168.202.154@tcp42 [8/256/0/180] [16257.387084] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16257.424484] LNet: Added LNI 192.168.202.154@tcp43 [8/256/0/180] [16266.977352] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16267.014736] LNet: Added LNI 192.168.202.154@tcp44 [8/256/0/180] [16276.915418] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16276.939898] LNet: Added LNI 192.168.202.154@tcp45 [8/256/0/180] [16287.278711] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16287.301157] LNet: Added LNI 192.168.202.154@tcp46 [8/256/0/180] [16297.000615] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16297.035103] LNet: Added LNI 192.168.202.154@tcp47 [8/256/0/180] [16305.518919] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16305.538797] LNet: Added LNI 192.168.202.154@tcp48 [8/256/0/180] [16313.933575] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16313.984193] LNet: Added LNI 192.168.202.154@tcp49 [8/256/0/180] [16323.143590] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16323.183849] LNet: Added LNI 192.168.202.154@tcp50 [8/256/0/180] [16346.085321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16346.105537] Lustre: Skipped 1 previous similar message [16346.111069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16346.135760] Lustre: Skipped 2 previous similar messages [16350.770484] Lustre: server umount lustre-MDT0000 complete [16355.350170] LustreError: 473454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857433 with bad export cookie 13921864048331247557 [16355.355211] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16355.366416] LustreError: 473454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16355.744881] Lustre: server umount lustre-MDT0001 complete [16372.709418] Lustre: 472733:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857434/real 1787857434] req@000000003487dfe4 x1874703959635840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857450 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16372.756331] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16372.786074] Lustre: Skipped 1 previous similar message [16375.862743] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [16376.799128] Lustre: 472734:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857438/real 1787857438] req@00000000d0e4abeb x1874703959635968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16377.826885] Lustre: 472736:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857439/real 1787857439] req@000000003e160ed6 x1874703959636096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16381.919257] Lustre: 472735:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857443/real 1787857443] req@00000000fc2bb7f3 x1874703959636352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16384.482704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16387.039195] Lustre: 472734:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857448/real 1787857448] req@00000000d9e4e863 x1874703959636480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16400.932848] LustreError: 475423:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16400.952264] LustreError: 475423:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [16401.024845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16402.019165] LustreError: 475424:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16402.027811] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16405.953922] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16407.523128] LustreError: 475423:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16412.644668] LustreError: 475424:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16414.750636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16415.065475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16418.152862] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16419.170057] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16427.127861] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16433.551440] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16451.044072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16451.061300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16451.081859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16452.065944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16454.117245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16454.128518] Lustre: Skipped 1 previous similar message [16459.257906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16459.266557] Lustre: Skipped 1 previous similar message [16464.359061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16465.378740] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16465.654462] Lustre: server umount lustre-OST0000 complete [16471.520679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16471.536500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16471.566334] Lustre: Skipped 1 previous similar message [16474.592286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16474.605883] Lustre: Skipped 3 previous similar messages [16476.954456] Lustre: server umount lustre-MDT0000 complete [16479.716592] LustreError: 499690:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16479.746307] LustreError: 499690:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16481.166556] LustreError: 473454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857558 with bad export cookie 13921864048331250133 [16481.169551] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16481.180438] LustreError: 473454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16481.479192] Lustre: server umount lustre-MDT0001 complete [16492.445788] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [16496.301527] Key type lgssc unregistered [16496.841377] LNet: 502474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16496.854545] LNetError: 502474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16496.863618] LNet: Removed LNI 192.168.202.154@tcp [16496.873762] LNet: Removed LNI 192.168.202.154@tcp10 [16496.880980] LNet: Removed LNI 192.168.202.154@tcp11 [16496.889766] LNet: Removed LNI 192.168.202.154@tcp12 [16496.896682] LNet: Removed LNI 192.168.202.154@tcp13 [16496.898492] LNet: Removed LNI 192.168.202.154@tcp14 [16496.902912] LNet: Removed LNI 192.168.202.154@tcp15 [16496.908717] LNet: Removed LNI 192.168.202.154@tcp16 [16496.919433] LNet: Removed LNI 192.168.202.154@tcp17 [16496.926193] LNet: Removed LNI 192.168.202.154@tcp18 [16496.935188] LNet: Removed LNI 192.168.202.154@tcp19 [16496.948582] LNet: Removed LNI 192.168.202.154@tcp20 [16496.952258] LNet: Removed LNI 192.168.202.154@tcp21 [16496.954911] LNet: Removed LNI 192.168.202.154@tcp22 [16496.961907] LNet: Removed LNI 192.168.202.154@tcp23 [16496.966182] LNet: Removed LNI 192.168.202.154@tcp24 [16496.975482] LNet: Removed LNI 192.168.202.154@tcp25 [16496.977425] LNet: Removed LNI 192.168.202.154@tcp26 [16496.988829] LNet: Removed LNI 192.168.202.154@tcp27 [16496.992086] LNet: Removed LNI 192.168.202.154@tcp28 [16496.997530] LNet: Removed LNI 192.168.202.154@tcp29 [16497.004853] LNet: Removed LNI 192.168.202.154@tcp30 [16497.009297] LNet: Removed LNI 192.168.202.154@tcp31 [16497.023692] LNet: Removed LNI 192.168.202.154@tcp32 [16497.028285] LNet: Removed LNI 192.168.202.154@tcp33 [16497.036243] LNet: Removed LNI 192.168.202.154@tcp34 [16497.044049] LNet: Removed LNI 192.168.202.154@tcp35 [16497.046882] LNet: Removed LNI 192.168.202.154@tcp36 [16497.053837] LNet: Removed LNI 192.168.202.154@tcp37 [16497.061431] LNet: Removed LNI 192.168.202.154@tcp38 [16497.067892] LNet: Removed LNI 192.168.202.154@tcp39 [16497.073631] LNet: Removed LNI 192.168.202.154@tcp40 [16497.078027] LNet: Removed LNI 192.168.202.154@tcp41 [16497.082270] LNet: Removed LNI 192.168.202.154@tcp42 [16497.086566] LNet: Removed LNI 192.168.202.154@tcp43 [16497.092601] LNet: Removed LNI 192.168.202.154@tcp44 [16497.098523] LNet: Removed LNI 192.168.202.154@tcp45 [16497.107423] LNet: Removed LNI 192.168.202.154@tcp46 [16497.109281] LNet: Removed LNI 192.168.202.154@tcp47 [16497.121389] LNet: Removed LNI 192.168.202.154@tcp48 [16497.126328] LNet: Removed LNI 192.168.202.154@tcp49 [16497.138826] LNet: Removed LNI 192.168.202.154@tcp50 [16498.183085] Key type .llcrypt unregistered [16498.189758] Key type ._llcrypt unregistered [16522.580391] Key type ._llcrypt registered [16522.582341] Key type .llcrypt registered [16522.671105] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [16537.673723] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [16538.652171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16538.945608] alg: No test for adler32 (adler32-zlib) [16540.084881] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [16540.251700] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [16541.936357] Key type lgssc registered [16542.983929] Lustre: Echo OBD driver; http://www.lustre.org/ [16581.749962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16581.798050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16583.294892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16583.332890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16583.534165] Lustre: lustre-MDT0000: new disk, initializing [16583.646552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16583.666540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16588.555835] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16600.110142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16600.360306] Lustre: 505675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16600.446198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16600.449812] Lustre: Skipped 1 previous similar message [16600.519454] Lustre: lustre-MDT0001: new disk, initializing [16600.643826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16600.706746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16600.716659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16605.221584] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16614.650626] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16620.274926] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16631.680905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16631.854365] Lustre: *** cfs_fail_loc=90f, val=0*** [16631.912342] Lustre: lustre-OST0000: new disk, initializing [16631.915916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16631.928445] Lustre: 507690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16632.015485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16638.005094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16638.021392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16638.174666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16638.759057] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16648.195661] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16658.913368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16658.918970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16658.925624] LustreError: Skipped 1 previous similar message [16658.925201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16658.942917] Lustre: Skipped 1 previous similar message [16664.035669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16664.043423] Lustre: Skipped 2 previous similar messages [16669.155652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16669.171370] Lustre: Skipped 1 previous similar message [16671.200032] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16671.433682] Lustre: server umount lustre-OST0000 complete [16677.344373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16677.353772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16677.372508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16679.394897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16682.285366] Lustre: server umount lustre-MDT0000 complete [16686.831375] LustreError: 508642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857764 with bad export cookie 17280694344236004211 [16686.835360] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16686.847199] LustreError: 508642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16687.303383] Lustre: server umount lustre-MDT0001 complete [16698.905851] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [16702.167512] Key type lgssc unregistered [16702.476928] LNet: 509600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16702.482028] LNetError: 509600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16702.500982] LNet: Removed LNI 192.168.202.154@tcp [16703.591187] Key type .llcrypt unregistered [16703.592712] Key type ._llcrypt unregistered [16725.786483] Key type ._llcrypt registered [16725.788241] Key type .llcrypt registered [16725.919769] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [16727.106988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16727.148665] alg: No test for adler32 (adler32-zlib) [16728.351184] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [16728.589660] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [16730.319210] Key type lgssc registered [16731.841267] Lustre: Echo OBD driver; http://www.lustre.org/ [16740.711783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16740.727850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16742.220482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16746.777725] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16752.609315] LustreError: 510875:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16752.634823] LustreError: 510875:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [16757.084514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16757.711727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16762.944309] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16772.165332] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16778.883165] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16786.164079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16786.617504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16794.272458] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16803.447646] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16814.231209] loop0: detected capacity change from 0 to 204800000 [16814.507926] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16814.514744] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16814.520047] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16821.257068] loop0: detected capacity change from 0 to 204800000 [16821.443276] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16821.463274] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16821.512307] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16828.919877] loop0: detected capacity change from 0 to 204800000 [16836.703362] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16836.886775] Lustre: fs15246-OST0000: new disk, initializing [16836.891202] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16836.898910] Lustre: 514469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16836.976831] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16844.157967] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16855.426156] loop1: detected capacity change from 0 to 204800000 [16861.014070] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16861.117409] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16861.176801] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16861.255433] Lustre: fs15246-MDT0000: new disk, initializing [16861.429309] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16861.481396] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16861.587557] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16862.624228] Lustre: 515591:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d2ed80e8 x1874704786834816/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787857956 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16865.683319] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [16871.799940] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16871.827720] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16871.869801] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16895.461500] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16895.469581] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16899.825694] Lustre: server umount fs15246-MDT0000 complete [16900.599378] LustreError: 512162:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16905.705409] LustreError: 510871:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16910.784249] Lustre: server umount fs15246-OST0000 complete [16926.197881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16926.199979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16926.247302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16931.335902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16931.335942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16931.356434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16931.389196] Lustre: Skipped 1 previous similar message [16932.805346] Lustre: server umount lustre-OST0000 complete [16941.540098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16941.542929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16941.572831] Lustre: Skipped 1 previous similar message [16943.787951] Lustre: server umount lustre-MDT0000 complete [16946.662742] LustreError: 510875:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16946.729087] LustreError: 510875:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16949.919691] LustreError: 510856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858027 with bad export cookie 13986555814341552075 [16949.933197] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16949.940220] LustreError: 510856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16950.758706] Lustre: server umount lustre-MDT0001 complete [16964.144660] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [16967.366891] Key type lgssc unregistered [16967.643368] LNet: 518692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16967.653851] LNetError: 518692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16967.672893] LNet: Removed LNI 192.168.202.154@tcp [16968.735982] Key type .llcrypt unregistered [16968.737648] Key type ._llcrypt unregistered [16992.432510] Key type ._llcrypt registered [16992.439967] Key type .llcrypt registered [16992.555741] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [16993.633085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16993.660205] alg: No test for adler32 (adler32-zlib) [16994.997826] Lustre: Lustre: Build Version: 2.17.57_97_gdd6e0cb [16995.358031] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [16997.015556] Key type lgssc registered [16998.417309] Lustre: Echo OBD driver; http://www.lustre.org/ [17045.849014] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [17056.596304] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [17098.938156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17098.973440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17100.256119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17100.281130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17100.363323] Lustre: lustre-MDT0000: new disk, initializing [17100.523289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17100.553388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17106.191266] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [17118.183704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17118.336288] Lustre: 523164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17118.394779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17118.397150] Lustre: Skipped 1 previous similar message [17118.526874] Lustre: lustre-MDT0001: new disk, initializing [17118.629074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17118.678397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17118.691438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17124.059397] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [17134.962218] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17141.462824] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17149.954136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17150.206479] Lustre: lustre-OST0000: new disk, initializing [17150.209994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17150.219800] Lustre: 525136:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17150.301125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17156.228148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17156.233589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17156.374596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17158.286456] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all [17169.417320] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17175.142344] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17175.372549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17181.035486] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17181.299569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17187.296252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17187.301885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17187.310692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17187.324517] Lustre: Skipped 1 previous similar message [17187.335955] Lustre: Skipped 1 previous similar message [17189.967932] Lustre: server umount lustre-OST0000 complete [17195.489805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17195.493685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17195.500527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17197.539957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17197.540272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17197.554650] Lustre: Skipped 1 previous similar message [17202.657413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17202.665054] Lustre: Skipped 1 previous similar message [17207.776696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17207.789474] Lustre: Skipped 1 previous similar message [17208.287234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17208.511492] Lustre: server umount lustre-MDT0000 complete [17213.091598] LustreError: 525146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858290 with bad export cookie 7544014132110006585 [17213.105694] LustreError: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17213.108848] LustreError: 525146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17213.640767] Lustre: server umount lustre-MDT0001 complete [17243.365511] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [17256.755086] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [17297.355182] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16951 sec ========= 15:19:33 (1787858373) [17299.494552] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:19:35 (1787858375) === [17304.762037] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:19:40 (1787858380) === [17355.941518] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [17359.229718] Key type lgssc unregistered [17359.523678] LNet: 531787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17359.537193] LNetError: 531787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17359.572669] LNet: Removed LNI 192.168.202.154@tcp [17360.904625] Key type .llcrypt unregistered [17360.907775] Key type ._llcrypt unregistered