[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471872883 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003175] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008629] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.012019] LSM: Security Framework initializing [ 0.013052] Yama: becoming mindful. [ 0.014035] SELinux: Initializing. [ 0.015075] *** VALIDATE selinux *** [ 0.023659] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027556] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030062] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031126] *** VALIDATE tmpfs *** [ 0.033280] *** VALIDATE proc *** [ 0.034262] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036231] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038124] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044329] debug: unmapping init [mem 0xffffffffb0259000-0xffffffffb0260fff] [ 0.046193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047620] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055278] rcu: Hierarchical SRCU implementation. [ 0.057338] smp: Bringing up secondary CPUs ... [ 0.058529] x86: Booting SMP configuration: [ 0.059038] .... node #0, CPUs: #1 #2 #3 [ 0.062112] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141795] node 0 deferred pages initialised in 73ms [ 0.143126] devtmpfs: initialized [ 0.144235] x86/mm: Memory block size: 128MB [ 0.146801] gcov: version magic: 0x41383552 [ 0.148287] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150239] pinctrl core: initialized pinctrl subsystem [ 0.151192] [ 0.151764] ************************************************************* [ 0.152015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153017] ** ** [ 0.154015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155014] ** ** [ 0.156015] ** This means that this kernel is built to expose internal ** [ 0.157015] ** IOMMU data structures, which may compromise security on ** [ 0.158016] ** your system. ** [ 0.159016] ** ** [ 0.160014] ** If you see this message and you are not debugging the ** [ 0.161014] ** kernel, report this immediately to your vendor! ** [ 0.162017] ** ** [ 0.163014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164018] ************************************************************* [ 0.165762] NET: Registered protocol family 16 [ 0.166474] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169427] cpuidle: using governor menu [ 0.171705] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174463] PCI: Using configuration type 1 for base access [ 0.176123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186159] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.189075] cryptd: max_cpu_qlen set to 1000 [ 0.191262] ACPI: Added _OSI(Module Device) [ 0.192014] ACPI: Added _OSI(Processor Device) [ 0.193012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198087] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200547] ACPI: Interpreter enabled [ 0.201083] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202017] ACPI: Using IOAPIC for interrupt routing [ 0.203113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.204398] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214158] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.216025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.217082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.219426] acpiphp: Slot [2] registered [ 0.220148] acpiphp: Slot [5] registered [ 0.221137] acpiphp: Slot [6] registered [ 0.222139] acpiphp: Slot [7] registered [ 0.223168] acpiphp: Slot [8] registered [ 0.224155] acpiphp: Slot [9] registered [ 0.225121] acpiphp: Slot [10] registered [ 0.226168] acpiphp: Slot [3] registered [ 0.227131] acpiphp: Slot [4] registered [ 0.228170] acpiphp: Slot [11] registered [ 0.229125] acpiphp: Slot [12] registered [ 0.230199] acpiphp: Slot [13] registered [ 0.231112] acpiphp: Slot [14] registered [ 0.232112] acpiphp: Slot [15] registered [ 0.233103] acpiphp: Slot [16] registered [ 0.234130] acpiphp: Slot [17] registered [ 0.235119] acpiphp: Slot [18] registered [ 0.236132] acpiphp: Slot [19] registered [ 0.237121] acpiphp: Slot [20] registered [ 0.238119] acpiphp: Slot [21] registered [ 0.239153] acpiphp: Slot [22] registered [ 0.240126] acpiphp: Slot [23] registered [ 0.241114] acpiphp: Slot [24] registered [ 0.242147] acpiphp: Slot [25] registered [ 0.243125] acpiphp: Slot [26] registered [ 0.244140] acpiphp: Slot [27] registered [ 0.245114] acpiphp: Slot [28] registered [ 0.246115] acpiphp: Slot [29] registered [ 0.247128] acpiphp: Slot [30] registered [ 0.248107] acpiphp: Slot [31] registered [ 0.249073] PCI host bridge to bus 0000:00 [ 0.250033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.251027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.252021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.253029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.254029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.255032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.256243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.257999] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.260324] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.266018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.270052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.271021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.272021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.273021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.274651] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.275767] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.276052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.277735] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.280015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.289869] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.291018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.295329] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.298017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.301018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.308021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.313891] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.343024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.355017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.382019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.393461] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.404019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.414016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.435020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.445078] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.452026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.457019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.471021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.479328] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.495017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.506020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.524016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.532000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.542019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.548013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.560029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.569279] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.572371] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.575345] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.577370] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.579224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.583188] iommu: Default domain type: Passthrough [ 0.585405] SCSI subsystem initialized [ 0.587172] ACPI: bus type USB registered [ 0.588105] usbcore: registered new interface driver usbfs [ 0.590102] usbcore: registered new interface driver hub [ 0.592082] usbcore: registered new device driver usb [ 0.593099] pps_core: LinuxPPS API ver. 1 registered [ 0.594019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.596046] PTP clock support registered [ 0.598059] EDAC MC: Ver: 3.0.0 [ 0.599109] PCI: Using ACPI for IRQ routing [ 0.600641] NetLabel: Initializing [ 0.602008] NetLabel: domain hash size = 128 [ 0.603010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.605088] NetLabel: unlabeled traffic allowed by default [ 0.606270] vgaarb: loaded [ 0.607282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.609015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.616000] clocksource: Switched to clocksource kvm-clock [ 0.703764] VFS: Disk quotas dquot_6.6.0 [ 0.704898] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.707201] *** VALIDATE ramfs *** [ 0.708429] *** VALIDATE hugetlbfs *** [ 0.709808] pnp: PnP ACPI init [ 0.711654] pnp: PnP ACPI: found 6 devices [ 0.728803] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.732318] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.734406] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.736582] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.739179] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.741927] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.744803] NET: Registered protocol family 2 [ 0.747141] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.751521] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.755015] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.759449] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.761733] TCP: Hash tables configured (established 65536 bind 65536) [ 0.763605] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.766306] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.768301] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.770248] NET: Registered protocol family 1 [ 0.771748] RPC: Registered named UNIX socket transport module. [ 0.773186] RPC: Registered udp transport module. [ 0.774393] RPC: Registered tcp transport module. [ 0.775703] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.777402] NET: Registered protocol family 44 [ 0.778743] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.782174] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.785173] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.788763] PCI: CLS 0 bytes, default 64 [ 0.791684] Unpacking initramfs... [ 2.150736] debug: unmapping init [mem 0xffff938dfcc54000-0xffff938dfffbffff] [ 2.156167] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.158200] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.160443] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.664140] Initialise system trusted keyrings [ 2.665393] Key type blacklist registered [ 2.666740] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.673097] zbud: loaded [ 2.675480] *** VALIDATE nfs *** [ 2.676295] *** VALIDATE nfs4 *** [ 2.677478] pstore: using deflate compression [ 2.680202] Platform Keyring initialized [ 2.779800] NET: Registered protocol family 38 [ 2.781373] Key type asymmetric registered [ 2.782703] Asymmetric key parser 'x509' registered [ 2.784401] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.787083] io scheduler mq-deadline registered [ 2.789163] io scheduler kyber registered [ 2.790932] io scheduler bfq registered [ 2.793839] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.797103] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.800158] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.803206] ACPI: Power Button [PWRF] [ 2.808648] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.815594] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.827340] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.835205] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.851629] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.879034] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.907225] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.912254] Non-volatile memory driver v1.3 [ 2.913680] Linux agpgart interface v0.103 [ 2.947156] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.950341] vda: detected capacity change from 0 to 76689408 [ 2.968846] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.971358] vdb: detected capacity change from 0 to 1073741824 [ 2.986421] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.990396] vdc: detected capacity change from 0 to 2621440000 [ 3.011980] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.015081] vdd: detected capacity change from 0 to 2621440000 [ 3.032059] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.034860] vde: detected capacity change from 0 to 4294967296 [ 3.050556] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.053374] vdf: detected capacity change from 0 to 4294967296 [ 3.066024] libphy: Fixed MDIO Bus: probed [ 3.072316] usbcore: registered new interface driver usbserial_generic [ 3.074226] usbserial: USB Serial support registered for generic [ 3.076057] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.079233] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.081267] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.084051] mousedev: PS/2 mouse device common for all mice [ 3.087407] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.087528] rtc_cmos 00:05: RTC can wake from S4 [ 3.096100] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.100347] rtc_cmos 00:05: registered as rtc0 [ 3.103848] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.104983] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.111803] intel_pstate: CPU model not supported [ 3.115250] hid: raw HID events driver (C) Jiri Kosina [ 3.117458] usbcore: registered new interface driver usbhid [ 3.119633] usbhid: USB HID core driver [ 3.121726] drop_monitor: Initializing network drop monitor service [ 3.123950] Initializing XFRM netlink socket [ 3.125927] NET: Registered protocol family 10 [ 3.128262] Segment Routing with IPv6 [ 3.129605] NET: Registered protocol family 17 [ 3.131699] mpls_gso: MPLS GSO support [ 3.137872] RAS: Correctable Errors collector initialized. [ 3.140161] AVX version of gcm_enc/dec engaged. [ 3.142046] AES CTR mode by8 optimization enabled [ 3.215729] sched_clock: Marking stable (3215708761, 0)->(4298046687, -1082337926) [ 3.220250] registered taskstats version 1 [ 3.222317] Loading compiled-in X.509 certificates [ 3.224833] zswap: loaded using pool lzo/zbud [ 3.249106] Key type big_key registered [ 3.261527] Key type encrypted registered [ 3.263596] ima: No TPM chip found, activating TPM-bypass! [ 3.265604] ima: Allocated hash algorithm: sha1 [ 3.267294] ima: No architecture policies found [ 3.269029] evm: Initialising EVM extended attributes: [ 3.271092] evm: security.selinux [ 3.272349] evm: security.ima [ 3.273479] evm: security.capability [ 3.274706] evm: HMAC attrs: 0x1 [ 3.276795] rtc_cmos 00:05: setting system clock to 2026-09-10 12:15:12 UTC (1789042512) [ 3.282291] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 3.285401] debug: unmapping init [mem 0xffffffffaff82000-0xffffffffb0258fff] [ 3.294102] Write protecting the kernel read-only data: 28672k [ 3.296663] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 3.298534] debug: unmapping init [mem 0xffffffffaef14000-0xffffffffaeffffff] [ 3.332051] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.338716] systemd[1]: Detected virtualization kvm. [ 3.340939] systemd[1]: Detected architecture x86-64. [ 3.342220] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.368610] systemd[1]: No hostname configured. [ 3.369937] systemd[1]: Set hostname to . [ 3.371724] random: systemd: uninitialized urandom read (16 bytes read) [ 3.373533] systemd[1]: Initializing machine ID from random generator. [ 3.500352] random: systemd: uninitialized urandom read (16 bytes read) [ 3.503072] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.506837] random: systemd: uninitialized urandom read (16 bytes read) [ 3.509463] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.514338] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.025581] device-mapper: uevent: version 1.0.3 [ 4.027160] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [[ 4.510079] random: fast init done  OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.669164] virtio_net virtio0 ens2: renamed from eth0 [ 4.730438] scsi host0: ata_piix [ 4.742100] scsi host1: ata_piix [ 4.743700] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.746220] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.120903] dracut-initqueue[575]: RTNETLINK answers: File exists [ 9.791227] random: crng init done [ 9.793555] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.199348] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.431615] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.708698] SELinux: Disabled at runtime. [ 11.772054] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.782241] systemd[1]: Detected virtualization kvm. [ 11.784167] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.280107] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.283703] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.288872] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.293337] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.296983] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.305788] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.313699] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK [ 12.333289] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.847552] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.207962] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.335509] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.472240] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.514336] EDAC sbridge: Ver: 1.1.2 [ 15.285695] Key type dns_resolver registered [ 15.622847] NFS: Registering the id_resolver key type [ 15.625578] Key type id_resolver registered [ 15.627643] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-server login: [ 51.176988] libcfs: loading out-of-tree module taints kernel. [ 51.231791] Key type ._llcrypt registered [ 51.234182] Key type .llcrypt registered [ 51.353158] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 72.561493] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 74.561625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 74.596415] alg: No test for adler32 (adler32-zlib) [ 76.410538] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 77.503169] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 79.279239] Key type lgssc registered [ 80.492368] hrtimer: interrupt took 4190338 ns [ 81.621131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 102.501246] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.080434] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 163.937947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 164.029015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.296459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.325845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.470756] Lustre: lustre-MDT0000: new disk, initializing [ 165.621155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.659044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.126504] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.908840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.042920] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 186.077622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 186.081905] Lustre: Skipped 1 previous similar message [ 186.285306] Lustre: lustre-MDT0001: new disk, initializing [ 186.490272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.563033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.590501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.731289] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.037161] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.395281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.556774] Lustre: lustre-OST0000: new disk, initializing [ 212.559421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.565831] Lustre: 8440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.624836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.912599] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.698919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.713342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.829971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.772260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.952710] Lustre: lustre-OST0001: new disk, initializing [ 236.964377] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.971574] Lustre: 9513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.063070] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.193494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 242.204226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 242.254071] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.963244] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.160910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.942215] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 270.538239] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing check_logdir /tmp/testlogs/ [ 276.718894] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing yml_node [ 282.061291] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 284.645854] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 287.368571] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 288.847661] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 08:19:56 EDT 2026 [ 305.432398] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 306.816074] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 324.075889] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.082330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.105507] Lustre: Skipped 1 previous similar message [ 324.132473] Lustre: Skipped 3 previous similar messages [ 329.188685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.397843] Lustre: server umount lustre-MDT0000 complete [ 335.014748] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789042844 with bad export cookie 329837720954594994 [ 335.020234] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.034879] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 339.424081] LustreError: 6510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.424856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.444195] LustreError: 6510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 339.473959] Lustre: Skipped 2 previous similar messages [ 339.476435] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.476440] Lustre: Skipped 3 previous similar messages [ 340.451993] LustreError: 10404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.460493] LustreError: 10404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 341.760892] Lustre: server umount lustre-MDT0001 complete [ 345.896672] Lustre: server umount lustre-OST0000 complete [ 349.384628] Lustre: server umount lustre-OST0001 complete [ 356.110487] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 365.660390] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 400.654103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.995690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 401.022248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.155255] Lustre: lustre-MDT0000: new disk, initializing [ 401.219203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.229764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.757319] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 419.460386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.693459] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 419.807160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 419.811797] Lustre: Skipped 1 previous similar message [ 419.965142] Lustre: lustre-MDT0001: new disk, initializing [ 420.065758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 420.147098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 420.156875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 427.145091] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 436.171778] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 445.231883] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 452.744478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 453.079307] Lustre: lustre-OST0000: new disk, initializing [ 453.082445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 453.090049] Lustre: 17036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 453.199194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.535913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 454.552479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 454.747472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 460.722787] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 470.736509] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 475.770494] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 475.999912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 481.268777] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 481.427441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 485.354613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.360148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.375092] LustreError: Skipped 1 previous similar message [ 485.400453] Lustre: Skipped 1 previous similar message [ 485.416687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.432409] Lustre: Skipped 4 previous similar messages [ 489.766108] Lustre: server umount lustre-OST0000 complete [ 495.588704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.593684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.617121] Lustre: Skipped 2 previous similar messages [ 500.556770] Lustre: server umount lustre-MDT0000 complete [ 500.704802] LustreError: 15078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.721150] LustreError: 15078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 504.174238] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789043013 with bad export cookie 329837720954596527 [ 504.179414] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.188636] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 504.528547] Lustre: server umount lustre-MDT0001 complete [ 509.685580] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:23:36 (1789043016) [ 512.816393] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 515.394568] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:23:42 (1789043022) [ 523.437989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.020314] LustreError: 19447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.085182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.391537] LustreError: 19448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.731360] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 539.621689] LustreError: 19448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.638244] LustreError: 19448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 539.837947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.844371] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 553.355986] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 558.761872] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.699550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 575.088873] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 583.410928] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 595.944403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 595.946716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.972403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 595.983700] Lustre: Skipped 1 previous similar message [ 596.012381] Lustre: Skipped 2 previous similar messages [ 601.334726] Lustre: server umount lustre-OST0000 complete [ 606.177081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.200464] Lustre: Skipped 1 previous similar message [ 611.927708] Lustre: server umount lustre-MDT0000 complete [ 616.181626] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789043125 with bad export cookie 329837720954597402 [ 616.199177] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.678616] Lustre: server umount lustre-MDT0001 complete [ 630.882902] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 634.532383] Key type lgssc unregistered [ 634.944811] LNet: 22983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.957496] LNetError: 22983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.986781] LNet: Removed LNI 192.168.202.159@tcp [ 636.190513] Key type .llcrypt unregistered [ 636.195293] Key type ._llcrypt unregistered [ 662.651274] Key type ._llcrypt registered [ 662.659656] Key type .llcrypt registered [ 662.785676] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 664.063470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 664.244391] alg: No test for adler32 (adler32-zlib) [ 665.599413] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 665.855029] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 667.599631] Key type lgssc registered [ 668.842126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 677.768726] loop: module loaded [ 677.885087] loop0: detected capacity change from 0 to 204800000 [ 678.197049] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.219629] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.703600] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.866347] loop0: detected capacity change from 0 to 204800000 [ 685.172973] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.193532] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.206115] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.878836] loop0: detected capacity change from 0 to 204800000 [ 697.722304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 697.773866] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.158984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 699.215117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 699.317471] Lustre: lustre-MDT0000: new disk, initializing [ 699.441924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.487967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 704.529830] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 712.941898] loop1: detected capacity change from 0 to 204800000 [ 718.503263] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 718.843269] Lustre: lustre-OST07c6: new disk, initializing [ 718.848441] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 718.852740] Lustre: Skipped 1 previous similar message [ 718.860390] Lustre: 26168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 718.935405] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 725.101896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 725.111795] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 725.150946] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 726.312754] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 731.017227] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 731.030045] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 731.037117] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 731.056771] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 731.065898] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 731.072032] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 749.535478] Lustre: 26774:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 749.547365] Lustre: 26774:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 749.552219] Lustre: 26774:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 749.561907] Lustre: 26774:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 749.573913] Lustre: 26774:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 749.578993] Lustre: 26774:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.042915] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 754.053138] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 754.062365] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 754.068632] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.076969] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.083328] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.093397] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 754.102667] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.112218] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 754.117544] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.123062] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.127445] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.326035] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 757.336643] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 757.346189] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 757.352615] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.358861] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.364917] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.370378] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 757.376253] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.386181] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 757.393660] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.401168] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 757.408123] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.362678] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 762.371882] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 762.382286] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.390821] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.398686] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.411883] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.424473] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 762.435283] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.445274] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 762.451607] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.461749] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.470241] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.110842] Lustre: 26774:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 772.132114] Lustre: 26774:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 772.156826] Lustre: 26774:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 772.179956] Lustre: 26774:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.214278] Lustre: 26774:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.240199] Lustre: 26774:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.255181] Lustre: 26774:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 772.275346] Lustre: 26774:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.300395] Lustre: 26774:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.318519] Lustre: 26774:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.333266] Lustre: 26774:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.349649] Lustre: 26774:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 827.361385] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.385599] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 830.594915] Lustre: server umount lustre-OST07c6 complete [ 841.357052] Lustre: server umount lustre-MDT0000 complete [ 860.064414] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 863.555373] Key type lgssc unregistered [ 863.767697] LNet: 28726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.774732] LNetError: 28726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.786836] LNet: Removed LNI 192.168.202.159@tcp [ 864.559112] Key type .llcrypt unregistered [ 864.561279] Key type ._llcrypt unregistered [ 888.056086] Key type ._llcrypt registered [ 888.060673] Key type .llcrypt registered [ 888.189197] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 890.093820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 890.181903] alg: No test for adler32 (adler32-zlib) [ 891.538663] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 891.940191] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 893.655388] Key type lgssc registered [ 895.192714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 904.958805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 904.978501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.604243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.020898] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 915.940758] LustreError: 29997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.959868] LustreError: 29997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 921.059793] LustreError: 29998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 921.364449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.691562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 926.524774] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 935.275429] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 940.657775] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 948.210091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 948.743071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.868455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 955.357470] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 963.015796] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 981.988260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 981.997058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.010546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 983.965860] Lustre: server umount lustre-OST0000 complete [ 988.134865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 988.144129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.165771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 992.224902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 992.234180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.354238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 997.368018] Lustre: Skipped 2 previous similar messages [ 998.566073] Lustre: server umount lustre-MDT0000 complete [ 1002.467419] LustreError: 29993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.504836] LustreError: 29993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1002.548879] LustreError: 29978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789043511 with bad export cookie 6011355112671709538 [ 1002.581792] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.986541] Lustre: server umount lustre-MDT0001 complete [ 1015.591821] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1019.333872] Key type lgssc unregistered [ 1019.705960] LNet: 33732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.711479] LNetError: 33732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.723271] LNet: Removed LNI 192.168.202.159@tcp [ 1021.069285] Key type .llcrypt unregistered [ 1021.075146] Key type ._llcrypt unregistered [ 1043.413912] Key type ._llcrypt registered [ 1043.417884] Key type .llcrypt registered [ 1043.659210] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1044.906783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1045.040117] alg: No test for adler32 (adler32-zlib) [ 1046.380345] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 1046.924848] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1048.639840] Key type lgssc registered [ 1050.674475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1062.252607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1062.263544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.800055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.852883] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1073.144310] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.176819] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1078.249931] LustreError: 35001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.420846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.893338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1083.783843] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1091.608984] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1100.025549] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.413445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.790433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.784405] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1124.148820] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1133.307447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1144.802846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1144.809215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.817063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1148.898194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.900950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1148.922350] Lustre: Skipped 1 previous similar message [ 1151.155508] Lustre: server umount lustre-OST0000 complete [ 1159.147697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.194380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.212456] Lustre: Skipped 1 previous similar message [ 1162.258268] Lustre: server umount lustre-MDT0000 complete [ 1164.257613] LustreError: 34995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.283538] LustreError: 34995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1166.910579] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789043676 with bad export cookie 16401323898958397595 [ 1166.912215] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.915799] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1167.272336] Lustre: server umount lustre-MDT0001 complete [ 1178.169832] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1181.611967] Key type lgssc unregistered [ 1182.028796] LNet: 38557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.040736] LNetError: 38557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.068256] LNet: Removed LNI 192.168.202.159@tcp [ 1183.062403] Key type .llcrypt unregistered [ 1183.065327] Key type ._llcrypt unregistered [ 1212.975489] Key type ._llcrypt registered [ 1212.978833] Key type .llcrypt registered [ 1213.214174] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1214.903607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1215.082446] alg: No test for adler32 (adler32-zlib) [ 1216.358750] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 1216.700543] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1218.399165] Key type lgssc registered [ 1219.861564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1228.902133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1228.917265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.563456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.022822] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1240.041503] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.082454] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1245.166180] LustreError: 39973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.471192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.908432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1252.413367] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1260.386637] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1266.467107] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1273.902744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1274.362992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.401637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1281.832250] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1290.021611] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1303.009944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1303.013353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.037137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1304.036842] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.041297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1304.077669] Lustre: Skipped 3 previous similar messages [ 1307.182497] Lustre: server umount lustre-MDT0000 complete [ 1308.401733] LustreError: 42092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.535343] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789043820 with bad export cookie 15641894496588651963 [ 1311.547527] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.558410] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1313.522971] LustreError: 39967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.548729] LustreError: 39967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1313.566926] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 1314.272369] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.284828] Lustre: Skipped 1 previous similar message [ 1316.335277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1316.342841] Lustre: Skipped 1 previous similar message [ 1317.983546] Lustre: server umount lustre-MDT0001 complete [ 1333.776641] Lustre: server umount lustre-OST0000 complete [ 1353.598582] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1356.194202] Key type lgssc unregistered [ 1356.503759] LNet: 43835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.511520] LNetError: 43835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.541316] LNet: Removed LNI 192.168.202.159@tcp [ 1358.372767] Key type .llcrypt unregistered [ 1358.375058] Key type ._llcrypt unregistered [ 1386.622471] Key type ._llcrypt registered [ 1386.624014] Key type .llcrypt registered [ 1386.783653] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1388.612617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1388.700131] alg: No test for adler32 (adler32-zlib) [ 1389.953174] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 1390.341528] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1392.159249] Key type lgssc registered [ 1393.406995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1401.445762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1401.485435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.053884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.882446] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1413.601940] LustreError: 45246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.613789] LustreError: 45246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1414.942795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.288586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1421.528859] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1431.001693] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1436.782958] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1444.576452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1445.027746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.137748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1453.125699] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1460.920172] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1468.897308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1468.903626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.913270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1470.946840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1470.956182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1470.963555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.995859] Lustre: Skipped 2 previous similar messages [ 1474.216690] Lustre: server umount lustre-OST0000 complete [ 1496.554915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.563101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.572903] Lustre: Skipped 1 previous similar message [ 1496.592606] Lustre: Skipped 1 previous similar message [ 1499.649244] Lustre: server umount lustre-MDT0000 complete [ 1501.666229] LustreError: 45246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.695274] LustreError: 45246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1503.705763] LustreError: 45227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789044012 with bad export cookie 5541808992565765524 [ 1503.710690] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1503.715948] LustreError: 45227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1518.054086] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1518.446614] Lustre: server umount lustre-MDT0001 complete [ 1532.450712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1535.683244] Key type lgssc unregistered [ 1536.056589] LNet: 48957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.063205] LNetError: 48957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.107143] LNet: Removed LNI 192.168.202.159@tcp [ 1537.223701] Key type .llcrypt unregistered [ 1537.225621] Key type ._llcrypt unregistered [ 1563.254290] Key type ._llcrypt registered [ 1563.255986] Key type .llcrypt registered [ 1563.433330] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1564.610642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1564.651281] alg: No test for adler32 (adler32-zlib) [ 1566.378101] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 1566.935655] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1568.711563] Key type lgssc registered [ 1570.957759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1582.510920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1582.556523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.169352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.706520] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1594.336846] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.355296] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1597.663878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.086793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1604.267918] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1613.101389] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1618.377828] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1625.703153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.966983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.059260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1632.298487] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1640.830798] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1647.035895] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1650.585695] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1663.125985] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-10 8h42m49s [ 1665.012142] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1669.105411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.116291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1669.129671] Lustre: Skipped 2 previous similar messages [ 1669.138751] Lustre: Skipped 1 previous similar message [ 1670.181550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1670.191687] Lustre: Skipped 4 previous similar messages [ 1673.598936] Lustre: server umount lustre-MDT0000 complete [ 1675.509400] LustreError: 50367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.475139] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1681.648589] LustreError: 50368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.661519] LustreError: 50368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1683.729444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.809230] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.962176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1683.992901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1684.975994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1690.034492] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1694.619317] Lustre: DEBUG MARKER: Wait for df (36689) ... [ 1697.072602] Lustre: DEBUG MARKER: done [ 1704.416347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1704.430890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.447501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1704.449697] Lustre: Skipped 1 previous similar message [ 1704.928921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.589467] Lustre: server umount lustre-OST0000 complete [ 1715.681419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.683081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1715.697925] Lustre: Skipped 1 previous similar message [ 1715.701405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1715.720702] Lustre: Skipped 4 previous similar messages [ 1719.354415] Lustre: server umount lustre-MDT0000 complete [ 1722.850820] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.878263] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1723.557767] LustreError: 52997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789044232 with bad export cookie 18097078344385209367 [ 1723.562431] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1723.573127] LustreError: 52997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1723.999505] Lustre: server umount lustre-MDT0001 complete [ 1733.290059] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1736.499568] Key type lgssc unregistered [ 1736.794680] LNet: 55296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.803907] LNetError: 55296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.823230] LNet: Removed LNI 192.168.202.159@tcp [ 1737.601296] Key type .llcrypt unregistered [ 1737.611338] Key type ._llcrypt unregistered [ 1778.721389] Key type ._llcrypt registered [ 1778.724184] Key type .llcrypt registered [ 1778.818803] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1780.391773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1780.478742] alg: No test for adler32 (adler32-zlib) [ 1781.708339] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 1781.976652] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1783.711176] Key type lgssc registered [ 1785.272954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1795.281813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1795.310413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.662573] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1796.685856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1796.923878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.489706] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1813.722799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.937215] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1814.021537] Lustre: 57500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1814.476177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1819.894033] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1829.227427] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1836.057523] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1843.506311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1843.776149] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1844.120983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.352758] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1853.807819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1862.811989] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1869.982608] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-10 8h46m17s [ 1873.206370] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1884.375945] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1888.490495] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1888.589056] LustreError: 57507:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1909.072826] Lustre: lustre-MDT0000: Client 9a1de4db-4090-4404-8818-cbc0f08d88eb (at 192.168.202.59@tcp) reconnecting [ 1928.623721] LustreError: 57507:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1931.328923] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1934.321334] Lustre: DEBUG MARKER: done [ 1940.960297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1940.966236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.975438] LustreError: Skipped 1 previous similar message [ 1940.981664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1941.001320] Lustre: Skipped 1 previous similar message [ 1941.038804] Lustre: Skipped 1 previous similar message [ 1942.267520] Lustre: server umount lustre-OST0000 complete [ 1951.205139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.229894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1951.247017] Lustre: Skipped 1 previous similar message [ 1951.274888] Lustre: Skipped 1 previous similar message [ 1952.355893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1952.384717] Lustre: Skipped 1 previous similar message [ 1954.663794] Lustre: server umount lustre-MDT0000 complete [ 1960.521772] LustreError: 57489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789044469 with bad export cookie 9768230169464077545 [ 1960.554922] LustreError: 57489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1960.591147] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1961.325711] Lustre: server umount lustre-MDT0001 complete [ 1975.922707] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1979.439292] Key type lgssc unregistered [ 1979.735982] LNet: 61834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.743471] LNetError: 61834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1979.768143] LNet: Removed LNI 192.168.202.159@tcp [ 1980.915463] Key type .llcrypt unregistered [ 1980.920435] Key type ._llcrypt unregistered [ 2033.125133] Key type ._llcrypt registered [ 2033.133410] Key type .llcrypt registered [ 2033.314554] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2034.839839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2034.891373] alg: No test for adler32 (adler32-zlib) [ 2036.061379] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 2036.455862] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2038.127203] Key type lgssc registered [ 2039.606324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2043.327762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2043.343344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.586967] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2044.608123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2044.841305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.798326] Lustre: Failing over lustre-MDT0000 [ 2046.045557] Lustre: server umount lustre-MDT0000 complete [ 2054.416388] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:49:21 (1789044561) [ 2061.982310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.609933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.991693] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2077.632976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.789556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2077.825403] Lustre: 64631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2078.116635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2083.071328] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2091.678624] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2099.077155] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2106.192302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2106.368582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2106.596598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.135736] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2114.731281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2122.426230] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2128.763766] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2139.616593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2139.627822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.655709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.641809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.644089] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.651927] Lustre: Skipped 1 previous similar message [ 2140.696815] Lustre: Skipped 1 previous similar message [ 2145.003912] Lustre: server umount lustre-MDT0000 complete [ 2149.305620] LustreError: 64624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789044658 with bad export cookie 15619842626481872940 [ 2149.306762] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.316316] LustreError: 64624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2149.573883] Lustre: server umount lustre-MDT0001 complete [ 2153.482119] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2158.572668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2167.221143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.263664] Lustre: 63555:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789044660/real 1789044660] req@00000000b8ee3d8e x1875949172860800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789044676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2167.311616] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.359201] Lustre: 63553:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789044664/real 1789044664] req@00000000986d731d x1875949172860928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789044680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2172.383264] Lustre: 63552:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789044665/real 1789044665] req@00000000e232873e x1875949172861056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789044681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2174.510067] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd8c4d41c7ef4e90f [ 2174.519616] Lustre: MGC192.168.202.159@tcp: Connection restored to 0@lo (at 0@lo) [ 2174.992254] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.023479] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2175.154111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.305981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2176.095807] Lustre: 63554:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789044668/real 1789044668] req@00000000701c587a x1875949172861312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789044684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2176.165219] LustreError: 66482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.197952] LustreError: 66482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2177.189723] LustreError: 68400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.217537] LustreError: 68400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2180.240409] Lustre: 63555:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789044673/real 1789044673] req@00000000a1bbb280 x1875949172861440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789044689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2182.116917] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2182.628587] LustreError: 68401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.674168] LustreError: 68401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2187.748296] LustreError: 68495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.779091] LustreError: 68495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2192.961072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.378479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.462052] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2198.458231] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2208.263525] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2215.580565] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2224.097443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2224.110558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.128847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2224.131562] Lustre: Skipped 1 previous similar message [ 2227.182433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.196826] Lustre: Skipped 2 previous similar messages [ 2229.075091] Lustre: server umount lustre-MDT0000 complete [ 2232.291368] LustreError: 68400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.312272] LustreError: 68400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2232.955330] LustreError: 64623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789044742 with bad export cookie 15619842626481875215 [ 2232.955867] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2232.962868] LustreError: 64623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2233.347907] Lustre: server umount lustre-MDT0001 complete [ 2239.916879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2249.474242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.791233] Lustre: 63555:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789044746/real 1789044746] req@00000000b546bb04 x1875949172891392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789044762 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2253.820552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.830659] Lustre: Skipped 2 previous similar messages [ 2258.247678] LustreError: 69138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.391236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.480860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2261.993905] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2270.478322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.623780] Lustre: 63554:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789044763/real 1789044763] req@00000000f420576b x1875949172892032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789044779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2270.659122] Lustre: 63554:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2275.576808] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2276.352110] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2276.376264] Lustre: Skipped 1 previous similar message [ 2282.550145] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2284.425585] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.320413] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2301.408414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2301.419552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.443963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.056152] Lustre: server umount lustre-MDT0000 complete [ 2308.072635] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.103611] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2308.928434] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789044818 with bad export cookie 15619842626481876594 [ 2308.931285] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2308.939604] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2309.510323] Lustre: server umount lustre-MDT0001 complete [ 2316.924469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2321.440792] Lustre: DEBUG MARKER: files compared the same [ 2333.066437] Lustre: server umount lustre-OST0000 complete [ 2351.559658] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2354.650806] Key type lgssc unregistered [ 2355.001831] LNet: 74362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.011530] LNetError: 74362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.074439] LNet: Removed LNI 192.168.202.159@tcp [ 2357.292370] Key type .llcrypt unregistered [ 2357.295764] Key type ._llcrypt unregistered [ 2381.777585] Key type ._llcrypt registered [ 2381.782441] Key type .llcrypt registered [ 2381.947889] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2383.081865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2383.168861] alg: No test for adler32 (adler32-zlib) [ 2384.674522] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 2384.885348] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2386.560069] Key type lgssc registered [ 2387.778993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2395.594227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2395.618776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.332513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.190699] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug +malloc all [ 2406.880553] LustreError: 75786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.895131] LustreError: 75786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2410.547442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.957791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2415.954968] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug +malloc all [ 2424.171626] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2429.387952] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2436.885621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2437.214537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.362223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2443.442369] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug +malloc all [ 2450.689637] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2467.296371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2467.304991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.317540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.339838] Lustre: Skipped 1 previous similar message [ 2469.867376] Lustre: server umount lustre-OST0000 complete [ 2477.537962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2477.542038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.551779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.551789] Lustre: Skipped 1 previous similar message [ 2477.579141] Lustre: Skipped 1 previous similar message [ 2479.744157] Lustre: server umount lustre-MDT0000 complete [ 2482.656545] LustreError: 75781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.675019] LustreError: 75781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2483.238729] LustreError: 75767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789044992 with bad export cookie 18130992698477642835 [ 2483.248596] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.253949] LustreError: 75767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2483.534433] Lustre: server umount lustre-MDT0001 complete [ 2494.089780] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2496.820433] Key type lgssc unregistered [ 2497.134160] LNet: 79343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2497.141163] LNetError: 79343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2497.169520] LNet: Removed LNI 192.168.202.159@tcp [ 2497.971289] Key type .llcrypt unregistered [ 2497.973334] Key type ._llcrypt unregistered [ 2523.658643] Key type ._llcrypt registered [ 2523.661487] Key type .llcrypt registered [ 2523.789924] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2524.648306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2524.730094] alg: No test for adler32 (adler32-zlib) [ 2525.760850] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 2526.067096] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2527.799819] Key type lgssc registered [ 2528.850517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2538.238443] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2538.275257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2554.080394] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2559.200109] LustreError: 80732:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 2585.887434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.601864] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2600.538675] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2608.607438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.729391] Lustre: *** cfs_fail_loc=706, val=0*** [ 2609.201864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.305274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2613.469484] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2614.247897] LustreError: 80757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.266666] LustreError: 80757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2619.377670] LustreError: 80758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.390677] LustreError: 81815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.430360] LustreError: 81815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2622.406269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.808278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2627.158404] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2634.794296] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2640.462567] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2643.433923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.439015] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2643.445760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2648.547100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2648.557979] Lustre: Skipped 2 previous similar messages [ 2648.857266] Lustre: server umount lustre-OST0000 complete [ 2653.668079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2653.679956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2653.684858] Lustre: Skipped 2 previous similar messages [ 2653.698676] Lustre: Skipped 1 previous similar message [ 2658.795370] LustreError: 81822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.825425] LustreError: 81822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2658.973834] Lustre: server umount lustre-MDT0000 complete [ 2663.103262] LustreError: 80738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789045172 with bad export cookie 15020822060055333631 [ 2663.110092] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2663.119080] LustreError: 80738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2663.431563] Lustre: server umount lustre-MDT0001 complete [ 2676.802183] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2680.001190] Key type lgssc unregistered [ 2680.368990] LNet: 84349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2680.376651] LNetError: 84349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2681.449885] LNet: Removed LNI 192.168.202.159@tcp [ 2682.195331] Key type .llcrypt unregistered [ 2682.198949] Key type ._llcrypt unregistered [ 2707.197831] Key type ._llcrypt registered [ 2707.200852] Key type .llcrypt registered [ 2707.332935] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2708.940252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2708.989466] alg: No test for adler32 (adler32-zlib) [ 2710.036444] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 2710.393802] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2712.111214] Key type lgssc registered [ 2713.476945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2721.955708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2721.987925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.434665] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2736.180269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.832771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2741.031838] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2742.250460] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.277130] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2746.339040] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.370149] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2750.504574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2750.882635] LustreError: 86453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.901425] LustreError: 86453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2750.932974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.065251] LustreError: 86449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.105530] LustreError: 86449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2758.750789] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2761.186450] LustreError: 86454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.204003] LustreError: 86454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2767.137991] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.160154] Lustre: MGS: Client b949eca5-4714-41ee-9cb5-ec4b52091a0b (at 0@lo) reconnecting [ 2767.187372] LustreError: 86464:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2767.509906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.592317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2772.434791] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2788.325961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2788.336416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.350688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2788.363678] Lustre: Skipped 1 previous similar message [ 2790.058968] Lustre: server umount lustre-OST0000 complete [ 2798.563367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.572538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2798.574759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2798.602571] Lustre: Skipped 2 previous similar messages [ 2803.685775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2803.692632] Lustre: Skipped 2 previous similar messages [ 2808.803249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.335251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2810.631214] Lustre: server umount lustre-MDT0000 complete [ 2813.925189] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.951071] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2814.736602] Lustre: server umount lustre-MDT0001 complete [ 2819.020572] Lustre: server umount MGS complete [ 2831.295583] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:02:19 (1789045339) [ 2862.356851] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2874.236277] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2911.723357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.403770] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2925.725690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.034440] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2926.121080] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2926.266828] Lustre: lustre-MDT0001: new disk, initializing [ 2926.342242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2926.372323] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2926.389441] Lustre: 93637:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2926.407402] LustreError: 93637:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2930.333028] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2940.407814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2940.670057] LustreError: 94387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.696784] Lustre: lustre-OST0000: new disk, initializing [ 2940.709996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2940.724652] Lustre: 94407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2940.793622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.042476] Lustre: 94406:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001330e07e x1875949879599616/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789045472 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2947.662822] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2956.926908] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2964.340618] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.355332] Lustre: MGS: Client e558da3e-e6d8-4308-ada3-9a8f1f537e74 (at 0@lo) reconnecting [ 2964.384594] LustreError: 93623:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2964.436114] Lustre: 92905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2964.506881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2964.619448] Lustre: lustre-MDT0000: new disk, initializing [ 2964.762323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.794623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2964.818521] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2965.856403] Lustre: 95473:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008a9faaf1 x1875949879606656/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789045491 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2970.018416] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2971.046465] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2974.703946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2974.720599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2975.130985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3000.802738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.809851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.829305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.833026] Lustre: Skipped 1 previous similar message [ 3006.923805] Lustre: server umount lustre-OST0000 complete [ 3012.576371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3012.579254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.585802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3012.594779] Lustre: Skipped 2 previous similar messages [ 3012.602838] Lustre: Skipped 3 previous similar messages [ 3017.698048] LustreError: 95896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.708391] Lustre: server umount lustre-MDT0000 complete [ 3017.711150] LustreError: 95896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3028.906461] Lustre: server umount lustre-MDT0001 complete [ 3032.816186] Lustre: server umount MGS complete [ 3045.567971] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:05:53 (1789045553) [ 3063.046619] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3066.164691] Key type lgssc unregistered [ 3066.465900] LNet: 97971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3066.469946] LNetError: 97971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3066.490602] LNet: Removed LNI 192.168.202.159@tcp [ 3067.440588] Key type .llcrypt unregistered [ 3067.442804] Key type ._llcrypt unregistered [ 3085.693379] Key type ._llcrypt registered [ 3085.696222] Key type .llcrypt registered [ 3085.879453] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3086.974455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3087.218172] alg: No test for adler32 (adler32-zlib) [ 3088.240320] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 3088.547626] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3090.255174] Key type lgssc registered [ 3091.257256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3099.032433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3099.045457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3100.489980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.173751] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3109.870908] LustreError: 99188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.896492] LustreError: 99188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3114.540257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.069440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3120.381820] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3128.909553] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3135.009734] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3142.053198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.483391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.590979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3149.220973] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3156.585453] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3170.275229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.277913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3170.288551] Lustre: Skipped 1 previous similar message [ 3170.313446] Lustre: Skipped 1 previous similar message [ 3173.929601] Lustre: server umount lustre-OST0000 complete [ 3180.513818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.535344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.539802] Lustre: Skipped 1 previous similar message [ 3180.563914] Lustre: Skipped 1 previous similar message [ 3184.757050] Lustre: server umount lustre-MDT0000 complete [ 3185.633274] LustreError: 99188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.647297] LustreError: 99188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3188.671251] LustreError: 99167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789045697 with bad export cookie 5638789332561494754 [ 3188.686760] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3189.147740] Lustre: server umount lustre-MDT0001 complete [ 3201.427887] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3203.715509] Key type lgssc unregistered [ 3204.557767] LNet: 102719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3204.569539] LNetError: 102719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3204.609415] LNet: Removed LNI 192.168.202.159@tcp [ 3206.211630] Key type .llcrypt unregistered [ 3206.213480] Key type ._llcrypt unregistered [ 3223.378261] Key type ._llcrypt registered [ 3223.390247] Key type .llcrypt registered [ 3223.567270] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3224.333154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3224.344053] alg: No test for adler32 (adler32-zlib) [ 3225.476382] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 3225.865811] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3227.599202] Key type lgssc registered [ 3229.060898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3235.230310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3235.239152] LustreError: 103797:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3235.274317] LustreError: 103801:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3235.291187] LustreError: 103797:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4968 [ 3235.294063] LustreError: 103797:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3235.301199] LustreError: 103797:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3235.315912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.048983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3246.967733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.380638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3252.908368] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3259.409430] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing lsmod [ 3262.184529] LustreError: 104991:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3262.302832] LustreError: 105007:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3262.307590] LustreError: 104991:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4884 [ 3262.311940] LustreError: 104991:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3262.323057] LustreError: 104991:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3262.325423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3262.745455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.810707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3268.064566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3268.067161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.077599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3269.090553] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.091153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3269.106823] Lustre: Skipped 1 previous similar message [ 3269.120342] Lustre: Skipped 2 previous similar messages [ 3272.004745] Lustre: server umount lustre-MDT0000 complete [ 3272.162615] LustreError: 103834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.203904] LustreError: 103834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3276.304407] LustreError: 103815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789045785 with bad export cookie 17802928360917839089 [ 3276.306839] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3276.327382] LustreError: 103815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3276.385874] LustreError: 104270:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bcaf5272 x1875950420644992/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3276.888600] Lustre: server umount lustre-MDT0001 complete [ 3291.447301] Lustre: server umount lustre-OST0000 complete [ 3299.149724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.796558] LustreError: 106072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.887822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3304.946067] LustreError: 106073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.891882] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3310.056447] LustreError: 106072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.765595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.236974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3321.444337] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3330.717432] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3338.349827] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3347.763080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.394365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.523287] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3364.332159] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3370.948863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3382.241781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3382.249345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.266029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3386.344728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3386.346978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3386.357629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.401942] Lustre: Skipped 1 previous similar message [ 3388.094317] Lustre: server umount lustre-OST0000 complete [ 3393.000827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3393.024147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.072106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.697674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.710814] Lustre: Skipped 3 previous similar messages [ 3407.329467] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3407.526295] Lustre: server umount lustre-MDT0000 complete [ 3410.922291] LustreError: 106073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.945025] LustreError: 106073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3411.900369] LustreError: 106053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789045921 with bad export cookie 17802928360917840020 [ 3411.942251] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3412.271182] Lustre: server umount lustre-MDT0001 complete [ 3425.369154] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3428.797320] Key type lgssc unregistered [ 3429.281805] LNet: 109630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3429.291431] LNetError: 109630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3429.312471] LNet: Removed LNI 192.168.202.159@tcp [ 3430.626473] Key type .llcrypt unregistered [ 3430.627962] Key type ._llcrypt unregistered [ 3458.369653] Key type ._llcrypt registered [ 3458.371648] Key type .llcrypt registered [ 3458.482944] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3459.559332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3459.613314] alg: No test for adler32 (adler32-zlib) [ 3460.994279] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 3461.188311] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3462.904121] Key type lgssc registered [ 3464.362293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3473.209806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3473.237060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.708411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.501932] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3484.133278] LustreError: 111050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3484.157522] LustreError: 111050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3489.255686] LustreError: 111051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3489.476682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.027183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3494.443608] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3502.405561] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3508.155484] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3515.743973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3516.100408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.199094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3523.903853] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3533.191513] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3541.765286] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3545.581843] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3550.176417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3550.179909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.206737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.200339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.206468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.210606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.245309] Lustre: Skipped 1 previous similar message [ 3555.336369] Lustre: server umount lustre-OST0000 complete [ 3561.441778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.447727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.458172] Lustre: Skipped 1 previous similar message [ 3561.483606] Lustre: Skipped 1 previous similar message [ 3565.890169] Lustre: server umount lustre-MDT0000 complete [ 3566.565080] LustreError: 111051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.618971] LustreError: 111051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3570.327241] LustreError: 111030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046079 with bad export cookie 14925082778113040389 [ 3570.356499] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3570.870318] Lustre: server umount lustre-MDT0001 complete [ 3583.349640] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3586.576481] Key type lgssc unregistered [ 3586.977792] LNet: 114689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3586.986891] LNetError: 114689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3587.004912] LNet: Removed LNI 192.168.202.159@tcp [ 3587.832221] Key type .llcrypt unregistered [ 3587.835792] Key type ._llcrypt unregistered [ 3606.110376] Key type ._llcrypt registered [ 3606.112913] Key type .llcrypt registered [ 3606.234762] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3607.153902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3607.291822] alg: No test for adler32 (adler32-zlib) [ 3608.452433] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 3608.752877] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3610.431270] Key type lgssc registered [ 3611.946484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3620.793065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3620.808815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.371715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.912676] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3632.611852] LustreError: 115910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.639535] LustreError: 115910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3637.730631] LustreError: 115909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3638.652604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.213291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3644.765580] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3656.544519] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3664.931996] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3674.396260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3675.116406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.118962] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3692.651223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3693.829726] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3704.801107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3704.820779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.826965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3704.855942] Lustre: Skipped 1 previous similar message [ 3704.880841] Lustre: Skipped 1 previous similar message [ 3707.878264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3707.897954] Lustre: Skipped 1 previous similar message [ 3709.892718] Lustre: server umount lustre-OST0000 complete [ 3716.084674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3716.107910] LustreError: Skipped 1 previous similar message [ 3716.117827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.153909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3718.114174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.250141] Lustre: server umount lustre-MDT0000 complete [ 3723.239933] LustreError: 115905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.265976] LustreError: 115905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3727.370351] LustreError: 117756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046236 with bad export cookie 17337451109137181545 [ 3727.397076] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3728.346574] Lustre: server umount lustre-MDT0001 complete [ 3741.950292] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3745.474846] Key type lgssc unregistered [ 3745.804810] LNet: 119445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3745.816928] LNetError: 119445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3745.835769] LNet: Removed LNI 192.168.202.159@tcp [ 3746.831240] Key type .llcrypt unregistered [ 3746.836648] Key type ._llcrypt unregistered [ 3776.572218] Key type ._llcrypt registered [ 3776.574449] Key type .llcrypt registered [ 3776.730824] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3778.338863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3778.554895] alg: No test for adler32 (adler32-zlib) [ 3779.743236] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 3780.231488] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3782.079201] Key type lgssc registered [ 3783.684820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3792.889644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3792.915447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.670892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3800.306493] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3804.134985] LustreError: 120863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3804.159415] LustreError: 120863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3809.250409] LustreError: 120864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3809.850240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.127328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3815.468235] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3824.234660] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3831.134143] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3838.222979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3838.539932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.544430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3845.438315] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3855.321876] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3862.204229] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3866.847896] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3866.864805] Lustre: Skipped 2 previous similar messages [ 3867.721514] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3867.785176] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3872.058595] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3872.071835] Lustre: Skipped 2 previous similar messages [ 3875.909723] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3875.913371] Lustre: Skipped 1 previous similar message [ 3883.592254] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.59@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3883.604548] Lustre: Skipped 2 previous similar messages [ 3891.730557] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.59@tcp 192.168.0.[2,10]@tcp [ 3896.365806] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.59/24@tcp in log lustre-MDT0000 [ 3896.375200] Lustre: Skipped 2 previous similar messages [ 3897.136294] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.59/24@tcp [ 3897.146585] Lustre: Skipped 1 previous similar message [ 3911.136883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3911.144660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3911.153571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3911.650427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3911.665438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3911.697695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3912.665267] Lustre: server umount lustre-OST0000 complete [ 3917.792963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3917.813425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.852419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3921.893607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3921.906718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.779506] Lustre: server umount lustre-MDT0000 complete [ 3927.010764] LustreError: 121841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.039213] LustreError: 121841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3929.278701] LustreError: 124204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046438 with bad export cookie 6067984875245587025 [ 3929.285340] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3929.289800] LustreError: 124204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3929.853879] Lustre: server umount lustre-MDT0001 complete [ 3944.081397] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3947.507542] Key type lgssc unregistered [ 3947.897960] LNet: 125734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3947.907914] LNetError: 125734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3947.933595] LNet: Removed LNI 192.168.202.159@tcp [ 3949.326095] Key type .llcrypt unregistered [ 3949.358553] Key type ._llcrypt unregistered [ 3984.041284] Key type ._llcrypt registered [ 3984.043623] Key type .llcrypt registered [ 3984.165302] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3985.309769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3985.417736] alg: No test for adler32 (adler32-zlib) [ 3986.521824] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 3986.780511] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3988.505473] Key type lgssc registered [ 3990.195279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3999.041909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3999.067069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4000.729738] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4000.753449] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.59/24@tcp [ 4000.807028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4006.418547] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4009.968756] LustreError: 127294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4009.992776] LustreError: 127294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4015.077749] LustreError: 127293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.202824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4016.546091] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4016.581214] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.59/24@tcp [ 4016.670838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4022.286591] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4030.541635] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4037.389921] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4045.667548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4046.174125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.709251] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4063.081874] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4070.779697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 4086.246513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4086.265290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.301565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.312085] Lustre: Skipped 1 previous similar message [ 4091.363462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4091.372767] Lustre: Skipped 1 previous similar message [ 4092.336449] Lustre: server umount lustre-OST0000 complete [ 4098.530663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4098.547677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.569405] Lustre: Skipped 1 previous similar message [ 4098.576478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4101.601819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4101.619029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4101.669917] Lustre: Skipped 1 previous similar message [ 4106.722220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4106.730782] Lustre: Skipped 1 previous similar message [ 4111.328147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4111.632437] Lustre: server umount lustre-MDT0000 complete [ 4111.842725] LustreError: 127293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.872697] LustreError: 127293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4115.939894] LustreError: 127294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.962655] LustreError: 127294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4116.281893] LustreError: 127274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046625 with bad export cookie 5777653984263683319 [ 4116.284578] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4116.302857] LustreError: 127274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4117.050053] Lustre: server umount lustre-MDT0001 complete [ 4131.408487] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4135.317776] Key type lgssc unregistered [ 4135.757633] LNet: 130907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4135.764285] LNetError: 130907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4135.779697] LNet: Removed LNI 192.168.202.159@tcp [ 4136.946545] Key type .llcrypt unregistered [ 4136.955720] Key type ._llcrypt unregistered [ 4167.473350] Key type ._llcrypt registered [ 4167.478942] Key type .llcrypt registered [ 4167.673824] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4184.593565] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4185.539917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4185.602087] alg: No test for adler32 (adler32-zlib) [ 4186.692933] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 4186.857778] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4188.495457] Key type lgssc registered [ 4190.029345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4232.608838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4232.625708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.024793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4234.069853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4234.175312] Lustre: lustre-MDT0000: new disk, initializing [ 4234.327877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.420263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4240.487809] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4254.486216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.646898] Lustre: 134308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4254.755775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4254.766552] Lustre: Skipped 1 previous similar message [ 4254.873582] Lustre: lustre-MDT0001: new disk, initializing [ 4255.062726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4255.130938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4255.142166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4261.346050] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4272.619242] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4279.386886] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4287.467409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4287.717306] Lustre: lustre-OST0000: new disk, initializing [ 4287.721248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4287.727201] Lustre: 136273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4287.835620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.830650] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4296.811653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4296.833551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4296.953373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4305.063138] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4311.009583] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4311.282801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4317.227507] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4317.551076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4322.785662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.789112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4322.826246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4322.845220] Lustre: Skipped 1 previous similar message [ 4326.290317] Lustre: server umount lustre-OST0000 complete [ 4332.013564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4332.031526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.056417] Lustre: Skipped 1 previous similar message [ 4332.059230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4333.038914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.169523] Lustre: server umount lustre-MDT0000 complete [ 4341.378880] LustreError: 134300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046850 with bad export cookie 40168699047051277 [ 4341.389969] LustreError: 134300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4341.391384] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4341.670814] Lustre: server umount lustre-MDT0001 complete [ 4352.332334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.881407] LustreError: 138490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4352.909823] LustreError: 138490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4352.987263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4358.116439] LustreError: 138491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.420377] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4363.234190] LustreError: 138490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.949097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.339082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4373.646838] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4381.419382] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4388.049369] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4395.362869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4395.860981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.584780] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4410.694983] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4416.224283] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4416.441646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4421.996223] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4422.338926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4432.554309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4432.739558] Lustre: lustre-OST0001: new disk, initializing [ 4432.743967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4432.750984] Lustre: 141865:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4432.841151] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4439.219356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4439.232854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4439.455777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4440.411393] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4449.844152] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4450.091839] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4455.653310] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4455.974487] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4462.374490] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9faf102fa800.ost_server_uuid 50 [ 4464.227650] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9faf102fa800.ost_server_uuid in IDLE state after 0 sec [ 4473.359511] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.371292] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4473.381151] Lustre: Skipped 1 previous similar message [ 4473.401399] Lustre: Skipped 3 previous similar messages [ 4478.447588] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4478.907507] Lustre: server umount lustre-OST0001 complete [ 4483.553628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.564354] LustreError: 140313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.566213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4483.566220] Lustre: Skipped 1 previous similar message [ 4483.569666] Lustre: Skipped 1 previous similar message [ 4483.601096] LustreError: 140313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4488.682288] LustreError: 140313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4488.696157] LustreError: 140313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4488.751196] Lustre: server umount lustre-OST0000 complete [ 4493.794034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.818201] Lustre: Skipped 1 previous similar message [ 4493.818430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4493.837695] Lustre: Skipped 2 previous similar messages [ 4498.329674] Lustre: server umount lustre-MDT0000 complete [ 4498.912535] LustreError: 138491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.929178] LustreError: 138491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4501.988481] LustreError: 138471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789047011 with bad export cookie 40168699047052159 [ 4501.998392] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4502.327722] Lustre: server umount lustre-MDT0001 complete [ 4523.804340] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4526.931902] Key type lgssc unregistered [ 4527.171759] LNet: 144868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4527.177600] LNetError: 144868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4527.191149] LNet: Removed LNI 192.168.202.159@tcp [ 4528.207593] Key type .llcrypt unregistered [ 4528.209746] Key type ._llcrypt unregistered [ 4575.870358] Key type ._llcrypt registered [ 4575.873177] Key type .llcrypt registered [ 4575.996168] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4591.413916] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4592.987267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4593.018190] alg: No test for adler32 (adler32-zlib) [ 4594.125562] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 4594.399026] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4596.144096] Key type lgssc registered [ 4597.604114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4633.438533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4633.459335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4634.751457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4634.770138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4634.838545] Lustre: lustre-MDT0000: new disk, initializing [ 4634.920123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4634.935120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4639.681087] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4651.576976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.684365] Lustre: 149023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4651.798470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4651.806481] Lustre: Skipped 1 previous similar message [ 4651.950362] Lustre: lustre-MDT0001: new disk, initializing [ 4652.086320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4652.159649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4652.176238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4656.754319] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4668.214321] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4674.053689] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4681.883123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4682.247631] Lustre: lustre-OST0000: new disk, initializing [ 4682.249853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4682.255473] Lustre: 150992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4682.357288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4687.984884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4688.000996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4688.129944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4689.069814] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4698.679317] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4702.950722] Lustre: Failing over lustre-OST0000 [ 4703.200990] Lustre: server umount lustre-OST0000 complete [ 4719.039112] Lustre: 147243:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789047212/real 1789047212] req@00000000e4f44fc5 x1875951855109504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789047228 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4719.066133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.166549] Lustre: 147246:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789047213/real 1789047213] req@00000000b56d8cf6 x1875951855110272/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789047229 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4720.202729] Lustre: 147246:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4721.702251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4721.972943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.990381] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4725.026540] Lustre: 147245:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789047218/real 1789047218] req@00000000706d472d x1875951855111168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789047234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4727.045816] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4728.126761] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4729.194616] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4729.196062] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4730.271099] Lustre: 147245:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789047223/real 1789047223] req@000000005f0d1dd6 x1875951855112192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789047239 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4730.287986] Lustre: 147245:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4733.615255] Lustre: Failing over lustre-MDT0000 [ 4733.921395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4733.937980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.951023] Lustre: Skipped 1 previous similar message [ 4734.310325] Lustre: server umount lustre-MDT0000 complete [ 4739.557539] LustreError: 150208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.591384] LustreError: 150208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4742.385838] LustreError: 149029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4744.673809] LustreError: 149035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4744.713751] LustreError: 149035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4747.518248] LustreError: 149828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.791538] Lustre: 147243:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789047243/real 1789047243] req@00000000c1c2c401 x1875951855122432/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789047259 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4749.854147] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4752.635422] LustreError: 149030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4752.683648] LustreError: 149030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4755.732306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.033190] LustreError: 147242:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000973c40c0 x1875951855127168/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4760.251110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4760.301620] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4762.861605] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4765.252331] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4765.674904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4765.678629] Lustre: Skipped 1 previous similar message [ 4765.687189] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4765.758693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4775.392550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4775.420824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.437994] Lustre: Skipped 2 previous similar messages [ 4775.452749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4779.993035] Lustre: server umount lustre-OST0000 complete [ 4786.145129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4786.150239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4786.152521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4786.187882] Lustre: Skipped 2 previous similar messages [ 4786.209306] Lustre: Skipped 2 previous similar messages [ 4790.789048] Lustre: server umount lustre-MDT0000 complete [ 4791.268848] LustreError: 150208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.292356] LustreError: 150208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4794.645578] LustreError: 149014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789047303 with bad export cookie 10797675937985768213 [ 4794.648908] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4794.655723] LustreError: 149014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4795.029596] Lustre: server umount lustre-MDT0001 complete [ 4808.640131] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4812.076895] Key type lgssc unregistered [ 4812.425657] LNet: 154650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4812.435351] LNetError: 154650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4812.462304] LNet: Removed LNI 192.168.202.159@tcp [ 4813.455759] Key type .llcrypt unregistered [ 4813.457416] Key type ._llcrypt unregistered [ 4853.598177] Key type ._llcrypt registered [ 4853.601480] Key type .llcrypt registered [ 4853.692040] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4854.941723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4855.222088] alg: No test for adler32 (adler32-zlib) [ 4856.419488] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 4856.687455] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4858.440495] Key type lgssc registered [ 4859.733628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4868.888154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4868.897933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.457243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4875.630850] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4879.841364] LustreError: 156169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.869057] LustreError: 156169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4884.960810] LustreError: 156170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4885.888347] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4889.996926] LustreError: 156198:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4890.010778] LustreError: 156198:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4890.020488] LustreError: 156198:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4890.461714] Lustre: server umount lustre-MDT0000 complete [ 4905.846560] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4915.838071] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4952.523328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.852582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4952.904158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4953.001838] Lustre: lustre-MDT0000: new disk, initializing [ 4953.148198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4953.167841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4958.643761] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4970.577751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.756669] Lustre: 159502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4970.841380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4970.852810] Lustre: Skipped 1 previous similar message [ 4970.974775] Lustre: lustre-MDT0001: new disk, initializing [ 4971.067446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4971.109067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4971.130666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4976.654662] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4986.348221] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4992.743168] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5000.781544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5001.062985] Lustre: lustre-OST0000: new disk, initializing [ 5001.070895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5001.075638] Lustre: 161471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5001.142358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.275378] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5009.512238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5009.523474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5009.700526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5019.703766] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5024.066772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5025.944512] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:38:53 (1789047533) [ 5265.890809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.908385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5265.912892] Lustre: Skipped 1 previous similar message [ 5265.926692] Lustre: Skipped 2 previous similar messages [ 5271.009272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5271.011571] Lustre: Skipped 1 previous similar message [ 5271.855755] Lustre: server umount lustre-MDT0000 complete [ 5275.563438] LustreError: 159493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789047784 with bad export cookie 14416451497050778017 [ 5275.564047] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.569578] LustreError: 159493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5276.127879] LustreError: 161064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5276.132622] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.148551] Lustre: Skipped 1 previous similar message [ 5276.151631] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5280.167756] LustreError: 159511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.171145] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5282.065262] Lustre: server umount lustre-MDT0001 complete [ 5292.850147] Lustre: server umount lustre-OST0000 complete [ 5304.493380] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5315.034723] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5350.546656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5350.707631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5350.734448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5350.848410] Lustre: lustre-MDT0000: new disk, initializing [ 5350.991984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5351.010748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5356.776663] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5368.134872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.289205] Lustre: 166138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5368.346927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5368.350356] Lustre: Skipped 1 previous similar message [ 5368.407285] Lustre: lustre-MDT0001: new disk, initializing [ 5368.488464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5368.520766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5368.535306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5374.054462] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5385.996419] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5393.157741] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5400.665482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5400.944696] Lustre: lustre-OST0000: new disk, initializing [ 5400.949883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5400.964884] Lustre: 168106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5401.065100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.802511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5406.820063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5406.961082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5408.295878] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5417.410955] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5422.487350] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5422.688525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5428.063728] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5428.376510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5432.801316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5432.806346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.814193] LustreError: Skipped 1 previous similar message [ 5432.821039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5432.839577] Lustre: Skipped 1 previous similar message [ 5432.859359] Lustre: Skipped 1 previous similar message [ 5436.548772] Lustre: server umount lustre-OST0000 complete [ 5443.058512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.089831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5443.132640] Lustre: Skipped 1 previous similar message [ 5448.162831] LustreError: 166147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.179775] LustreError: 166147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5448.285672] Lustre: server umount lustre-MDT0000 complete [ 5452.967473] LustreError: 166131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789047962 with bad export cookie 14416451497050936847 [ 5452.998231] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5453.492280] Lustre: server umount lustre-MDT0001 complete [ 5462.271488] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:46:10 (1789047970) [ 5480.812338] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5492.947242] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5534.049145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5534.327165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5534.352725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5534.432157] Lustre: lustre-MDT0000: new disk, initializing [ 5534.612361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.643853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5541.121228] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5553.007512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.177164] Lustre: 172866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5553.552502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5558.452626] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5568.236556] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5573.764331] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5581.169601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5581.606658] Lustre: lustre-OST0000: new disk, initializing [ 5581.612020] Lustre: Skipped 1 previous similar message [ 5581.622426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5581.634151] Lustre: Skipped 2 previous similar messages [ 5581.640863] Lustre: 174844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5588.905022] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5591.121326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5591.125762] Lustre: Skipped 1 previous similar message [ 5591.132974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5591.233921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5600.302806] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5612.002686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5612.005360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.017635] LustreError: Skipped 1 previous similar message [ 5612.023638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5612.045352] Lustre: Skipped 1 previous similar message [ 5612.077835] Lustre: Skipped 2 previous similar messages [ 5614.911309] Lustre: server umount lustre-OST0000 complete [ 5620.193159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5620.200107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.388782] Lustre: server umount lustre-MDT0000 complete [ 5627.373398] LustreError: 174033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.390941] LustreError: 174033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5629.614197] LustreError: 174843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048138 with bad export cookie 14416451497050937729 [ 5629.622377] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5629.630948] LustreError: 174843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5630.029267] Lustre: server umount lustre-MDT0001 complete [ 5638.518644] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:49:06 (1789048146) [ 5655.132820] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5666.489962] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5708.326780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.648814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5708.673850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5708.739859] Lustre: lustre-MDT0000: new disk, initializing [ 5708.811871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5708.816917] Lustre: Skipped 2 previous similar messages [ 5708.830245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5714.263783] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5725.179828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5725.338879] Lustre: 179167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5725.553050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5730.458874] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5739.396218] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5745.644621] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5752.582766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5753.069327] Lustre: 181147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5758.670255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5758.950949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5760.412558] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5771.937245] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5779.428695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.433634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5779.444243] Lustre: Skipped 2 previous similar messages [ 5779.450489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5779.450498] Lustre: Skipped 3 previous similar messages [ 5779.487872] LustreError: Skipped 1 previous similar message [ 5785.442606] Lustre: server umount lustre-OST0000 complete [ 5795.865619] Lustre: server umount lustre-MDT0000 complete [ 5799.915917] LustreError: 179177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.939739] LustreError: 179177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5799.965272] LustreError: 179161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048309 with bad export cookie 14416451497050938814 [ 5799.968394] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5799.977048] LustreError: 179161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5800.306875] Lustre: server umount lustre-MDT0001 complete [ 5813.598560] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5817.170332] Key type lgssc unregistered [ 5817.628113] LNet: 183002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5817.637740] LNetError: 183002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5817.655271] LNet: Removed LNI 192.168.202.159@tcp [ 5818.825099] Key type .llcrypt unregistered [ 5818.826878] Key type ._llcrypt unregistered [ 5847.264506] Key type ._llcrypt registered [ 5847.265929] Key type .llcrypt registered [ 5847.411954] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5848.340018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5848.371233] alg: No test for adler32 (adler32-zlib) [ 5849.619558] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 5849.834620] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5851.543309] Key type lgssc registered [ 5852.728915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5861.679865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5861.714264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5863.177135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.763056] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5873.635414] LustreError: 184417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.651341] LustreError: 184417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5877.696411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.276108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5883.923545] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5891.745165] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5897.410779] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5904.705566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5905.221155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5908.340087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5911.943675] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5919.990422] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5944.297093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5944.304619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.321263] LustreError: Skipped 1 previous similar message [ 5944.331980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5944.342493] Lustre: Skipped 1 previous similar message [ 5944.363359] Lustre: Skipped 1 previous similar message [ 5948.358619] Lustre: server umount lustre-OST0000 complete [ 5954.528466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5954.546106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5959.650527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5959.658028] Lustre: Skipped 2 previous similar messages [ 5964.773255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5964.783801] Lustre: Skipped 1 previous similar message [ 5967.327508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5967.571312] Lustre: server umount lustre-MDT0000 complete [ 5969.891518] LustreError: 184417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5969.917755] LustreError: 184417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5972.564226] LustreError: 184398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048481 with bad export cookie 2639675473118710203 [ 5972.580285] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5973.148811] Lustre: server umount lustre-MDT0001 complete [ 5986.344821] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5989.562578] Key type lgssc unregistered [ 5989.969139] LNet: 187988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5989.980458] LNetError: 187988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5990.009493] LNet: Removed LNI 192.168.202.159@tcp [ 5990.950879] Key type .llcrypt unregistered [ 5990.953416] Key type ._llcrypt unregistered [ 6016.379764] Key type ._llcrypt registered [ 6016.385500] Key type .llcrypt registered [ 6016.531660] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6017.755074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6017.855149] alg: No test for adler32 (adler32-zlib) [ 6019.117336] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 6019.462894] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6021.114566] Key type lgssc registered [ 6022.756028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6031.639605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6031.669928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6033.287000] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.618570] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6043.620991] LustreError: 189404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.651160] LustreError: 189404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6047.582058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.912247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6051.956318] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6060.692308] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6069.057252] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6076.738035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6077.013416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6078.119746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6082.636943] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6090.397642] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6102.289394] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 6104.038544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6104.042665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.062764] LustreError: Skipped 1 previous similar message [ 6104.091370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6107.379263] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 6107.395456] Lustre: Skipped 1 previous similar message [ 6112.523580] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 6112.530544] Lustre: Skipped 2 previous similar messages [ 6113.247325] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6113.396968] Lustre: server umount lustre-OST0000 complete [ 6120.706580] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9faf0759d000.ost_server_uuid 50 [ 6122.205426] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6139.894977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6139.907486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6139.919381] Lustre: Skipped 2 previous similar messages [ 6139.937689] Lustre: Skipped 1 previous similar message [ 6142.340287] Lustre: server umount lustre-MDT0000 complete [ 6144.994783] LustreError: 190748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.026181] LustreError: 190748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6146.800610] LustreError: 192349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048656 with bad export cookie 3213644079265720794 [ 6146.805359] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6146.809626] LustreError: 192349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6147.158728] Lustre: server umount lustre-MDT0001 complete [ 6155.570256] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:57:43 (1789048663) [ 6163.974444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.615592] LustreError: 193319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6164.696466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.092433] LustreError: 193320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.925937] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6174.189114] LustreError: 193319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.878790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6183.341782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6189.106675] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6197.215795] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6204.179167] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6211.926483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6212.390502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.092393] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6229.205537] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6233.965482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6238.160757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6238.452760] Lustre: lustre-OST0001: new disk, initializing [ 6238.457782] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6238.468836] Lustre: 196161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6238.628734] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6247.146489] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6248.181139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6248.194464] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6248.391285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6257.855720] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6263.573037] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 6263.784834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6267.050809] Lustre: server umount lustre-OST0000 complete [ 6268.658385] LustreError: 195143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6268.694094] LustreError: 195143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6273.932894] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6274.188372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6279.498280] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6279.744782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6289.165759] LustreError: 195141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.194408] LustreError: 195141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6294.497974] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6294.502281] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6294.512312] LustreError: Skipped 1 previous similar message [ 6294.531201] Lustre: Skipped 1 previous similar message [ 6294.533394] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6294.555874] Lustre: Skipped 3 previous similar messages [ 6300.465696] Lustre: server umount lustre-OST0001 complete [ 6304.743084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.801972] Lustre: Skipped 1 previous similar message [ 6311.232101] Lustre: server umount lustre-MDT0000 complete [ 6316.243267] LustreError: 193300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048825 with bad export cookie 3213644079265721865 [ 6316.247274] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6316.253851] LustreError: 193300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6316.901238] Lustre: server umount lustre-MDT0001 complete [ 6343.283363] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:00:51 (1789048851) [ 6350.068549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.533630] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6350.597115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6350.924425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.832835] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6366.979981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.179542] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6367.309261] Lustre: 199602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6372.700401] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6380.703984] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6386.753887] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6394.214189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6394.432915] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6401.752730] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6401.771217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6409.595695] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.845404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.955190] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6418.218519] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6418.225935] Lustre: Skipped 2 previous similar messages [ 6424.726589] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6425.311417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6433.892950] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6437.345396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6437.361129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.372630] Lustre: Skipped 1 previous similar message [ 6437.376406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.380462] Lustre: Skipped 6 previous similar messages [ 6442.927129] Lustre: server umount lustre-OST0000 complete [ 6444.798467] LustreError: 202254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.823555] LustreError: 202254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6461.412209] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6461.435616] LustreError: Skipped 1 previous similar message [ 6461.446584] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.461725] Lustre: Skipped 1 previous similar message [ 6464.309637] Lustre: server umount lustre-OST0001 complete [ 6476.945254] Lustre: server umount lustre-MDT0000 complete [ 6482.280723] LustreError: 202500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048991 with bad export cookie 3213644079265723272 [ 6482.284806] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6482.292668] LustreError: 202500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6482.705584] Lustre: server umount lustre-MDT0001 complete [ 6509.281403] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:03:36 (1789049016) [ 6527.442677] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 6538.966336] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6576.372542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6576.739861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6576.768345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6576.859421] Lustre: lustre-MDT0000: new disk, initializing [ 6576.937335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6576.952816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6582.103819] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6593.408716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.537330] Lustre: 207711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6593.586812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6593.590733] Lustre: Skipped 1 previous similar message [ 6593.647819] Lustre: lustre-MDT0001: new disk, initializing [ 6593.792958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6593.807342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6599.365941] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6610.392479] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6617.964856] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6626.335941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6626.607571] Lustre: lustre-OST0000: new disk, initializing [ 6626.612328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6626.618210] Lustre: 209681:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6634.318968] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6635.103747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6635.116214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6635.291968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6644.711325] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6649.912750] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6650.227922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6655.845381] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6656.049649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6661.093727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6661.101559] LustreError: Skipped 1 previous similar message [ 6661.106944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6661.113979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6661.119823] Lustre: Skipped 4 previous similar messages [ 6661.143325] Lustre: Skipped 9 previous similar messages [ 6664.825368] Lustre: server umount lustre-OST0000 complete [ 6670.816672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6670.822975] LustreError: Skipped 1 previous similar message [ 6675.557168] Lustre: server umount lustre-MDT0000 complete [ 6676.452081] LustreError: 207723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.485883] LustreError: 207723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6680.210132] LustreError: 209679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049189 with bad export cookie 3213644079265725057 [ 6680.219873] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6680.222171] LustreError: 209679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6680.649353] Lustre: server umount lustre-MDT0001 complete [ 6690.976520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6697.525996] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6707.255031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.877598] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6721.204810] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6727.564497] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6735.246244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6742.849967] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6750.181834] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6755.218441] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6755.421237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6760.794954] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6761.113776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6767.584643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6767.587041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.592116] LustreError: Skipped 1 previous similar message [ 6767.611816] Lustre: Skipped 3 previous similar messages [ 6769.535939] Lustre: server umount lustre-OST0000 complete [ 6775.507091] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6780.866100] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6811.223037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6819.021990] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6827.736158] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6830.379873] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6832.328446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6832.346872] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6832.355440] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6842.852517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6842.863698] Lustre: Skipped 6 previous similar messages [ 6852.067747] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6852.287765] Lustre: server umount lustre-OST0000 complete [ 6863.329360] LustreError: 211895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6863.347564] LustreError: 211895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6863.441622] Lustre: server umount lustre-MDT0000 complete [ 6868.573138] LustreError: 212636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049377 with bad export cookie 3213644079265725939 [ 6868.574972] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6868.582562] LustreError: 212636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6869.113584] Lustre: server umount lustre-MDT0001 complete [ 6880.377948] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:09:47 (1789049387) [ 6888.791287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.453321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6889.466193] Lustre: Skipped 6 previous similar messages [ 6895.726714] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6905.943126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.468869] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6922.173560] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6928.563572] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6937.063653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6945.194990] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6953.693548] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6959.619415] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6959.915300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6965.214899] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6965.515711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6972.481675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6972.777971] Lustre: lustre-OST0001: new disk, initializing [ 6972.785834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6972.789923] Lustre: 221244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6978.697295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6978.718631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6978.865825] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6981.259408] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6993.470599] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6999.597440] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6999.871763] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7005.884438] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7006.147341] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7009.768248] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7009.769606] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7009.803709] Lustre: Skipped 5 previous similar messages [ 7014.676224] Lustre: server umount lustre-OST0001 complete [ 7022.726695] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7023.022452] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7028.549824] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7028.854744] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7059.632289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7065.064787] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7065.075144] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7065.078842] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7065.104875] Lustre: Skipped 2 previous similar messages [ 7067.262271] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7075.004141] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7076.655498] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7080.417285] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7080.436184] LustreError: Skipped 1 previous similar message [ 7093.218565] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7093.395251] Lustre: server umount lustre-OST0001 complete [ 7112.676210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.694965] Lustre: Skipped 12 previous similar messages [ 7117.904060] Lustre: server umount lustre-OST0000 complete [ 7128.836583] Lustre: server umount lustre-MDT0000 complete [ 7131.616479] LustreError: 217872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7131.641928] LustreError: 217872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7132.913374] LustreError: 217852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049642 with bad export cookie 3213644079265726905 [ 7132.918916] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7132.924682] LustreError: 217852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7139.567287] Lustre: server umount lustre-MDT0001 complete [ 7168.260198] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:14:35 (1789049675) [ 7175.992744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.419541] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7176.486064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7182.341296] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7193.442127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.555776] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7193.641766] Lustre: 226491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7199.646193] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7208.616308] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7216.776617] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7225.707221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7225.984624] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7233.707698] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7243.476630] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7253.837840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7253.984670] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7263.289468] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7270.232987] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7272.091852] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7276.536943] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7276.922162] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7282.055372] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7282.455072] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7290.016730] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9faf10aef000.ost_server_uuid 50 [ 7291.904674] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9faf10aef000.ost_server_uuid in IDLE state after 0 sec [ 7293.201497] Lustre: Permanently deactivating lustre-OST0001 [ 7293.248420] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7299.051922] Lustre: Permanently reactivating lustre-OST0001 [ 7299.076858] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7299.083089] Lustre: Skipped 2 previous similar messages [ 7305.206340] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7305.221788] LustreError: Skipped 3 previous similar messages [ 7305.230729] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7305.256935] Lustre: Skipped 6 previous similar messages [ 7310.197157] Lustre: server umount lustre-OST0001 complete [ 7320.292786] Lustre: server umount lustre-OST0000 complete [ 7332.120382] Lustre: server umount lustre-MDT0000 complete [ 7336.528968] LustreError: 230903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049845 with bad export cookie 3213644079265728193 [ 7336.537162] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7336.537898] LustreError: 230903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7336.927535] Lustre: server umount lustre-MDT0001 complete [ 7365.843090] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:17:53 (1789049873) [ 7374.531047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7374.922053] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7374.959813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7374.968715] Lustre: Skipped 2 previous similar messages [ 7380.388778] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7389.329763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7389.464656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7389.502931] Lustre: 233194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7395.607288] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7403.927407] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7410.298889] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7417.582213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7417.790143] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7417.810212] Lustre: Permanently deactivating lustre-OST0000 [ 7417.821347] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7418.003197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.007947] Lustre: Skipped 10 previous similar messages [ 7424.279677] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7426.121935] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7431.433833] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7438.468266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7438.597060] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7444.217623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7446.338234] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7455.440807] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7457.488669] Lustre: Permanently reactivating lustre-OST0000 [ 7457.543424] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7457.552743] Lustre: Skipped 2 previous similar messages [ 7464.104059] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7464.142036] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 38 seconds. I think it's dead, and I am evicting it. exp 00000000fa6a26e3, cur 1789049973 deadline 1789049965 last 1789049935 [ 7464.219074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7481.073222] Lustre: server umount lustre-OST0001 complete [ 7500.259112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7500.518832] Lustre: server umount lustre-OST0000 complete [ 7512.258054] Lustre: server umount lustre-MDT0000 complete [ 7517.435172] LustreError: 233186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050026 with bad export cookie 3213644079265729859 [ 7517.445394] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7517.802191] Lustre: server umount lustre-MDT0001 complete [ 7532.503416] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7536.397779] Key type lgssc unregistered [ 7536.805823] LNet: 238073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7536.817501] LNetError: 238073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7536.851503] LNet: Removed LNI 192.168.202.159@tcp [ 7538.313088] Key type .llcrypt unregistered [ 7538.318433] Key type ._llcrypt unregistered [ 7566.306470] Key type ._llcrypt registered [ 7566.308246] Key type .llcrypt registered [ 7566.440429] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7567.926156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7568.009851] alg: No test for adler32 (adler32-zlib) [ 7569.234626] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 7569.600214] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7571.407366] Key type lgssc registered [ 7572.755428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7583.307167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7583.338849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.068966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.517828] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7595.493530] LustreError: 239542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7595.501971] LustreError: 239542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7600.391665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7600.541661] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7600.571044] Lustre: 239530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7600.581549] Lustre: Permanently deactivating lustre-MDT0001 [ 7600.590843] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7600.610298] LustreError: 239541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7600.955640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7606.412364] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7607.920551] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7614.806608] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7621.085603] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7629.865430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7630.384964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.404707] LustreError: 241384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7631.452820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7636.461290] LustreError: 241384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7636.888468] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7641.572974] LustreError: 241382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.581121] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7646.711045] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7651.786284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7651.814944] LustreError: 241385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7651.832582] LustreError: 241385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7652.073411] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7657.469024] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7657.471298] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7657.626546] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7665.971845] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7669.177554] Lustre: Permanently reactivating lustre-MDT0001 [ 7669.191270] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7669.195249] Lustre: Skipped 2 previous similar messages [ 7674.144779] Lustre: lustre-MDT0001: Client dbaba878-ad6b-4a58-b043-0126faa6c628 (at 192.168.202.59@tcp) reconnecting [ 7679.056719] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7679.070545] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7679.077691] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7679.088642] LustreError: 243258:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7679.096953] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7679.098593] LustreError: 243258:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7679.098720] LustreError: 243258:0:(obd_config.c:2200:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -108 [ 7679.099088] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7679.099088] [ 7679.099566] LustreError: 239532:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7680.711773] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7682.483677] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7688.343892] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7688.582568] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7692.563820] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7692.806896] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7695.657243] Lustre: Permanently deactivating lustre-MDT0001 [ 7695.668918] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7695.672061] Lustre: Skipped 2 previous similar messages [ 7701.055441] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7710.255160] Lustre: server umount lustre-MDT0000 complete [ 7713.764952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7713.779832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.800648] LustreError: 239541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7714.788536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7714.818159] Lustre: Skipped 2 previous similar messages [ 7715.026697] LustreError: 243094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050224 with bad export cookie 11049511206831339854 [ 7715.030421] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7715.047939] LustreError: 243094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7715.429161] Lustre: server umount lustre-MDT0001 complete [ 7720.004263] Lustre: server umount lustre-OST0000 complete [ 7733.002603] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:24:00 (1789050240) [ 7746.486319] Lustre: server umount lustre-OST0001 complete [ 7755.250045] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 7766.491728] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7806.860877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7807.285601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7807.302286] Lustre: Skipped 2 previous similar messages [ 7807.336466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7807.394616] Lustre: lustre-MDT0000: new disk, initializing [ 7807.444142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7807.465794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7813.951751] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7824.763641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.935423] Lustre: 247803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7824.991929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7824.995198] Lustre: Skipped 1 previous similar message [ 7825.077279] Lustre: lustre-MDT0001: new disk, initializing [ 7825.192526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7825.239876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7825.249133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7830.976224] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7841.137220] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7847.526830] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7855.275562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7855.822578] Lustre: lustre-OST0000: new disk, initializing [ 7855.829538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7855.836659] Lustre: 249774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7855.988950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7861.874313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7861.884521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7861.985711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7863.457883] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7875.856180] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7877.778684] LustreError: 247812:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7886.617870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.841468] Lustre: lustre-OST0001: new disk, initializing [ 7886.847984] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7886.857083] Lustre: 250888:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7887.807079] LustreError: 247812:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7887.838649] LustreError: 247810:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7893.683420] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7897.243683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7897.258703] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7897.413436] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7897.879152] LustreError: 247810:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7904.681734] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7906.650792] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7911.719354] Lustre: lustre-OST0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 7912.937185] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7912.951871] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7912.955273] LustreError: Skipped 1 previous similar message [ 7912.971143] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7912.982444] Lustre: Skipped 1 previous similar message [ 7913.006043] Lustre: Skipped 1 previous similar message [ 7915.005727] Lustre: server umount lustre-OST0001 complete [ 7916.784775] LustreError: 249757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.811897] LustreError: 249757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7933.425944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.438929] Lustre: Skipped 1 previous similar message [ 7933.444880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.460904] Lustre: Skipped 1 previous similar message [ 7938.529946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7938.547040] Lustre: Skipped 1 previous similar message [ 7938.877575] Lustre: server umount lustre-OST0000 complete [ 7943.649836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.664848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7948.257761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7949.766288] Lustre: server umount lustre-MDT0000 complete [ 7953.638603] LustreError: 247796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050462 with bad export cookie 11049511206831342668 [ 7953.645678] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7953.649785] LustreError: 247796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7960.158423] Lustre: server umount lustre-MDT0001 complete [ 7971.897758] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7974.639941] Key type lgssc unregistered [ 7974.980678] LNet: 252924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7974.989422] LNetError: 252924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7975.000701] LNet: Removed LNI 192.168.202.159@tcp [ 7975.947804] Key type .llcrypt unregistered [ 7975.950591] Key type ._llcrypt unregistered [ 8016.549586] Key type ._llcrypt registered [ 8016.553240] Key type .llcrypt registered [ 8016.659606] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8018.416832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8018.443243] alg: No test for adler32 (adler32-zlib) [ 8019.685659] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 8019.927970] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8021.631163] Key type lgssc registered [ 8022.965717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8030.998505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8031.020231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8032.268408] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8032.291080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8032.490616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8037.759090] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8046.164272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8046.275829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8046.325898] Lustre: 255130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8046.689810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8051.490304] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8058.682650] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8064.084864] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8070.383767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8070.588798] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8070.776661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8076.976735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8077.356858] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8084.288554] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8138.219334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8138.230497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.245322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8138.724465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8138.734394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.754099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8140.062632] Lustre: server umount lustre-OST0000 complete [ 8145.593165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8158.744093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8159.211197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8160.225877] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8160.265868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8160.278380] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8166.211169] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8174.858892] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8216.566509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8216.567339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.576363] LustreError: Skipped 1 previous similar message [ 8216.577110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8216.591675] Lustre: Skipped 1 previous similar message [ 8216.615946] Lustre: Skipped 1 previous similar message [ 8220.296986] Lustre: server umount lustre-OST0000 complete [ 8225.760916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8225.770245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8225.777142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8231.906836] LustreError: 255142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.933376] LustreError: 255142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8232.015722] Lustre: server umount lustre-MDT0000 complete [ 8235.658376] LustreError: 255871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050744 with bad export cookie 11861469026770241134 [ 8235.662487] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8235.671254] LustreError: 255871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8236.268927] Lustre: server umount lustre-MDT0001 complete [ 8247.170578] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8250.598318] Key type lgssc unregistered [ 8250.916257] LNet: 260553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8250.925846] LNetError: 260553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8250.939339] LNet: Removed LNI 192.168.202.159@tcp [ 8251.947536] Key type .llcrypt unregistered [ 8251.951460] Key type ._llcrypt unregistered [ 8277.240297] Key type ._llcrypt registered [ 8277.241994] Key type .llcrypt registered [ 8277.353914] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8278.745247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8279.017784] alg: No test for adler32 (adler32-zlib) [ 8280.161838] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 8280.375265] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8282.097101] Key type lgssc registered [ 8283.269287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8291.652744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8291.698904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.247197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.884490] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8303.594463] LustreError: 261965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8303.630166] LustreError: 261965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8306.989437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8307.379455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8312.503804] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8320.161606] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8325.007422] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8332.592785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8332.977771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.120978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8340.773544] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8349.783263] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8378.340652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8378.347253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.354742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8378.381674] Lustre: Skipped 1 previous similar message [ 8380.922957] Lustre: server umount lustre-OST0000 complete [ 8388.577126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8388.580612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8388.599456] Lustre: Skipped 1 previous similar message [ 8388.611206] Lustre: Skipped 2 previous similar messages [ 8393.698616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8393.709439] Lustre: Skipped 1 previous similar message [ 8397.794220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8397.810740] Lustre: Skipped 1 previous similar message [ 8398.303259] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8398.820625] Lustre: server umount lustre-MDT0000 complete [ 8402.400708] LustreError: 261945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050911 with bad export cookie 1815215066076872098 [ 8402.413958] LustreError: 261945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8402.420305] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8402.653843] Lustre: server umount lustre-MDT0001 complete [ 8412.072546] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8414.719836] Key type lgssc unregistered [ 8414.954682] LNet: 266272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8414.974816] LNetError: 266272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8416.045467] LNet: Removed LNI 192.168.202.159@tcp [ 8417.238291] Key type .llcrypt unregistered [ 8417.240308] Key type ._llcrypt unregistered [ 8437.168528] Key type ._llcrypt registered [ 8437.171551] Key type .llcrypt registered [ 8437.313125] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8438.327927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8438.480861] alg: No test for adler32 (adler32-zlib) [ 8439.637447] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 8439.865434] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8441.599201] Key type lgssc registered [ 8442.633568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8450.173746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8450.198213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8451.729543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8456.424421] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8461.288796] LustreError: 267490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8461.313460] LustreError: 267490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8464.668070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.993070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8469.292791] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8476.711654] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8482.960890] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8489.999658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8490.394029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.147330] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8496.631694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8503.638854] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8517.095119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8517.106965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8517.110037] LustreError: Skipped 1 previous similar message [ 8517.110982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8517.130109] Lustre: Skipped 1 previous similar message [ 8517.148856] Lustre: Skipped 1 previous similar message [ 8519.299801] Lustre: server umount lustre-OST0000 complete [ 8526.305580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8526.312296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.340099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8527.339190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8527.372386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8527.374822] Lustre: Skipped 1 previous similar message [ 8531.885746] Lustre: server umount lustre-MDT0000 complete [ 8532.470690] LustreError: 267486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8532.498804] LustreError: 267486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8535.852569] LustreError: 268224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051045 with bad export cookie 4643226268668827133 [ 8535.867247] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8535.870744] LustreError: 268224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8536.265859] Lustre: server umount lustre-MDT0001 complete [ 8547.175642] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8550.335444] Key type lgssc unregistered [ 8550.669667] LNet: 271166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8550.677255] LNetError: 271166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8550.703175] LNet: Removed LNI 192.168.202.159@tcp [ 8551.895789] Key type .llcrypt unregistered [ 8551.899119] Key type ._llcrypt unregistered [ 8575.313568] Key type ._llcrypt registered [ 8575.316422] Key type .llcrypt registered [ 8575.428884] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8576.511615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8576.560957] alg: No test for adler32 (adler32-zlib) [ 8578.085843] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 8578.351255] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8580.127209] Key type lgssc registered [ 8581.669676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8588.937727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8588.957292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.349538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.706795] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8600.546362] LustreError: 272578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.567893] LustreError: 272578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8602.941454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.305289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8607.957799] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8616.882428] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8622.666396] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8629.254394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8629.581512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.699520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8635.975638] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8642.723742] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8676.832430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8676.837934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.856549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8678.894312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8678.897268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8678.919524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8680.762931] Lustre: server umount lustre-OST0000 complete [ 8685.024836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8685.042436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.072401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8685.078675] Lustre: Skipped 1 previous similar message [ 8689.124609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8689.127348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8690.719278] Lustre: server umount lustre-MDT0000 complete [ 8694.250413] LustreError: 275018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8694.264291] LustreError: 275018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8695.326744] LustreError: 272559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051204 with bad export cookie 4844435984050589470 [ 8695.351317] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8695.354386] LustreError: 272559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8695.934083] Lustre: server umount lustre-MDT0001 complete [ 8710.164395] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8714.039556] Key type lgssc unregistered [ 8714.470443] LNet: 276883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8714.484426] LNetError: 276883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8714.507690] LNet: Removed LNI 192.168.202.159@tcp [ 8715.845634] Key type .llcrypt unregistered [ 8715.849540] Key type ._llcrypt unregistered [ 8737.711200] Key type ._llcrypt registered [ 8737.713167] Key type .llcrypt registered [ 8737.968975] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8739.573321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8739.766723] alg: No test for adler32 (adler32-zlib) [ 8740.950778] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 8741.309878] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8743.007359] Key type lgssc registered [ 8744.735553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8753.725794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8753.753773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8755.458906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8760.896969] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8764.913446] LustreError: 278118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8764.931325] LustreError: 278118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8770.032614] LustreError: 278120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8770.552820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8770.990042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8776.182258] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8784.017720] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8789.551898] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8796.877729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8797.238671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8800.306666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8804.777982] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8813.118918] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8830.950728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8830.966082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.981504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8831.971217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8831.973402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8831.974384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.978619] Lustre: Skipped 1 previous similar message [ 8835.725367] Lustre: server umount lustre-OST0000 complete [ 8842.209303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8842.229681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8842.236921] Lustre: Skipped 1 previous similar message [ 8846.285517] Lustre: server umount lustre-MDT0000 complete [ 8847.330373] LustreError: 278120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8847.357263] LustreError: 278120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8850.576718] LustreError: 278084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051359 with bad export cookie 3815631108784621591 [ 8850.582308] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8850.587845] LustreError: 278084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8850.879937] Lustre: server umount lustre-MDT0001 complete [ 8861.966711] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8865.106252] Key type lgssc unregistered [ 8865.348103] LNet: 281821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8865.361340] LNetError: 281821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8865.398864] LNet: Removed LNI 192.168.202.159@tcp [ 8866.299075] Key type .llcrypt unregistered [ 8866.300456] Key type ._llcrypt unregistered [ 8880.701864] Key type ._llcrypt registered [ 8880.705102] Key type .llcrypt registered [ 8880.813726] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8900.923068] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8915.851671] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8916.616874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8916.637386] alg: No test for adler32 (adler32-zlib) [ 8917.707811] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 8918.062967] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8919.791413] Key type lgssc registered [ 8921.185241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8960.570032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8960.607717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8961.987894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8962.051809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8962.274232] Lustre: lustre-MDT0000: new disk, initializing [ 8962.487628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8962.512336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8968.223165] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8980.892590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8980.979192] Lustre: 285374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8981.037578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8981.041127] Lustre: Skipped 1 previous similar message [ 8981.133986] Lustre: lustre-MDT0001: new disk, initializing [ 8981.221143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8981.255964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8981.274497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8985.568289] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8994.177362] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8999.270554] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9006.185563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9006.532751] Lustre: lustre-OST0000: new disk, initializing [ 9006.536719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9006.546697] Lustre: 287341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9006.654514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9013.308805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9013.328256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9013.455491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9013.844294] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9024.238103] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9029.563564] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9029.782817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9034.513911] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9034.740208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9039.331285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9039.340482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9039.373314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9039.384942] Lustre: Skipped 1 previous similar message [ 9042.904691] Lustre: server umount lustre-OST0000 complete [ 9048.033859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9048.060830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.101508] Lustre: Skipped 1 previous similar message [ 9048.111087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9049.573267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9049.578069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.688703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.698824] Lustre: Skipped 2 previous similar messages [ 9058.789578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9058.791296] Lustre: Skipped 1 previous similar message [ 9062.367836] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9062.660367] Lustre: server umount lustre-MDT0000 complete [ 9067.205084] LustreError: 285364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051576 with bad export cookie 2402042320966338267 [ 9067.227842] LustreError: 285364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9067.230770] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9067.972237] Lustre: server umount lustre-MDT0001 complete [ 9080.562068] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:46:27 (1789051587) [ 9087.828750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9088.520681] LustreError: 289757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9088.545571] LustreError: 289757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9088.621391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9093.605530] LustreError: 289758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9093.852250] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9098.723135] LustreError: 289757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9102.539194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9103.109070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9108.612705] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9116.664680] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9122.432579] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9129.724869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9130.001107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9135.679224] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9142.679378] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9161.169958] Lustre: server umount lustre-OST0000 complete [ 9168.864905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.879288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9168.888211] Lustre: Skipped 1 previous similar message [ 9168.905614] Lustre: Skipped 1 previous similar message [ 9176.178956] Lustre: server umount lustre-MDT0000 complete [ 9179.103484] Lustre: 283594:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789051672/real 1789051672] req@000000004efbb0b4 x1875956389193600/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789051688 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9179.109498] LustreError: 289758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.134889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9179.156310] LustreError: 289758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9180.421607] LustreError: 292260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051689 with bad export cookie 2402042320966339149 [ 9180.423927] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9180.433142] LustreError: 292260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9180.836339] Lustre: server umount lustre-MDT0001 complete [ 9192.019643] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9195.231359] Key type lgssc unregistered [ 9195.593602] LNet: 293312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9195.620940] LNetError: 293312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9196.657480] LNet: Removed LNI 192.168.202.159@tcp [ 9197.500900] Key type .llcrypt unregistered [ 9197.503069] Key type ._llcrypt unregistered [ 9221.039201] Key type ._llcrypt registered [ 9221.046558] Key type .llcrypt registered [ 9221.162878] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 9235.291310] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9236.026798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9236.079025] alg: No test for adler32 (adler32-zlib) [ 9237.043915] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 9237.279988] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9238.951800] Key type lgssc registered [ 9239.918330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9280.660362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9280.683372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.842412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9281.902755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9282.064299] Lustre: lustre-MDT0000: new disk, initializing [ 9282.261501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9282.357140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9287.159247] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9298.309847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.492630] Lustre: 297151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9298.560419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9298.565388] Lustre: Skipped 1 previous similar message [ 9298.646968] Lustre: lustre-MDT0001: new disk, initializing [ 9298.736904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9298.796552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9298.818316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9304.003584] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9313.031913] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9317.969294] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9324.258366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9324.584619] Lustre: lustre-OST2710: new disk, initializing [ 9324.587956] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9324.596603] Lustre: 299119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9324.675146] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9330.574713] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9334.867489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9334.873853] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9334.964426] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9340.331074] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9347.901870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9348.119386] Lustre: lustre-OST03e8: new disk, initializing [ 9348.124424] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9348.132310] Lustre: 300182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9348.250707] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9354.917010] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9354.928732] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9355.105099] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9355.465411] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9364.504726] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9372.078979] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9372.384418] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9377.933546] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9378.285722] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9383.207528] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9383.483262] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9388.443135] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9388.783802] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9395.995169] Lustre: server umount lustre-MDT0000 complete [ 9396.201605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9396.220285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9399.688563] LustreError: 297143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051908 with bad export cookie 16035964479005811050 [ 9399.710303] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9399.713274] LustreError: 297143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9400.219612] Lustre: server umount lustre-MDT0001 complete [ 9415.043028] Lustre: server umount lustre-OST2710 complete [ 9417.567152] Lustre: 294948:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789051910/real 1789051910] req@0000000009dcc85d x1875956723682688/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789051926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9417.597829] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9417.620140] Lustre: Skipped 3 previous similar messages [ 9419.319418] Lustre: server umount lustre-OST03e8 complete [ 9437.914432] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 9446.872754] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9482.409711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9482.585253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9482.606628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9482.667807] Lustre: lustre-MDT0000: new disk, initializing [ 9482.733332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9482.745638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9486.459830] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9496.043401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9496.200504] Lustre: 305431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9496.258825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9496.265916] Lustre: Skipped 1 previous similar message [ 9496.365916] Lustre: lustre-MDT0001: new disk, initializing [ 9496.474153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9496.537391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9496.549623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9501.049057] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9510.725783] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9515.940386] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9523.867632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9524.362832] Lustre: lustre-OST0000: new disk, initializing [ 9524.368215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9524.378730] Lustre: 307399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9524.495691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.071330] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9533.603064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9533.624726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9533.740263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9541.903343] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9547.572815] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9547.800183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9552.844498] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9553.133446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9559.521822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9559.527342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9559.533543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9559.569194] Lustre: Skipped 1 previous similar message [ 9561.539922] Lustre: server umount lustre-OST0000 complete [ 9568.224253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9568.230515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9568.242115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9568.244519] Lustre: Skipped 1 previous similar message [ 9569.762539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9569.768525] Lustre: Skipped 1 previous similar message [ 9571.993052] Lustre: server umount lustre-MDT0000 complete [ 9574.888116] LustreError: 305438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9574.911362] LustreError: 305438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9575.376514] LustreError: 306226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052084 with bad export cookie 16035964479005812730 [ 9575.381538] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9575.382945] LustreError: 306226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9575.789560] Lustre: server umount lustre-MDT0001 complete [ 9584.466270] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:54:51 (1789052091) [ 9586.212958] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9587.920759] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:54:55 (1789052095) [ 9591.988727] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9618.100790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9618.360600] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9618.381663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9618.488021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9623.390189] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9633.368090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9633.494158] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9633.515708] Lustre: 311258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9638.078947] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9659.770676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9660.064325] Lustre: Denying initial registration attempt from nid 192.168.202.159@tcp, specified as failover [ 9660.070187] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9660.076763] LustreError: 312712:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9660.081144] LustreError: 312712:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9660.083483] LustreError: 312712:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9660.131755] Lustre: server umount lustre-OST0000 complete [ 9660.133770] LustreError: 312712:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9667.553176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9667.562154] Lustre: Skipped 1 previous similar message [ 9667.566274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9667.573238] Lustre: Skipped 1 previous similar message [ 9672.673249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9672.676019] Lustre: Skipped 1 previous similar message [ 9673.065299] Lustre: server umount lustre-MDT0000 complete [ 9677.792680] LustreError: 311264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9677.819088] LustreError: 311264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9677.866721] LustreError: 311249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052187 with bad export cookie 16035964479005813612 [ 9677.882412] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9677.890771] LustreError: 311249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9678.224603] Lustre: server umount lustre-MDT0001 complete [ 9689.705763] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9694.094385] Key type lgssc unregistered [ 9694.531209] LNet: 313832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9694.545445] LNetError: 313832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9694.565059] LNet: Removed LNI 192.168.202.159@tcp [ 9695.802525] Key type .llcrypt unregistered [ 9695.804641] Key type ._llcrypt unregistered [ 9706.221701] Key type ._llcrypt registered [ 9706.222866] Key type .llcrypt registered [ 9706.295640] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9707.090376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9707.204676] alg: No test for adler32 (adler32-zlib) [ 9708.696644] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 9709.160410] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9710.999222] Key type lgssc registered [ 9712.373186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9760.122353] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9769.382235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9769.394519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9770.710933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9770.754902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9770.922788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9776.189548] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9784.884629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9785.029295] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9785.091924] Lustre: 316593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9785.388543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9790.140225] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9797.419351] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9803.877641] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9823.023831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9823.392134] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9823.645288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9831.308171] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9840.182736] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9845.218265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9845.227752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.256786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9845.734821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.736402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9848.439905] Lustre: server umount lustre-OST0000 complete [ 9851.875261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9851.882408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9851.894972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9851.898909] Lustre: Skipped 1 previous similar message [ 9855.971334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9855.972686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9857.701402] Lustre: server umount lustre-MDT0000 complete [ 9862.162816] LustreError: 316584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052371 with bad export cookie 511929187748069066 [ 9862.175699] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9862.177831] LustreError: 316584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9862.730959] Lustre: server umount lustre-MDT0001 complete [ 9870.596200] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9873.457792] Key type lgssc unregistered [ 9873.790574] LNet: 320163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9873.795316] LNetError: 320163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9873.812823] LNet: Removed LNI 192.168.202.159@tcp [ 9874.777105] Key type .llcrypt unregistered [ 9874.780777] Key type ._llcrypt unregistered [ 9902.820563] Key type ._llcrypt registered [ 9902.824318] Key type .llcrypt registered [ 9902.939782] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9904.231452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9904.393557] alg: No test for adler32 (adler32-zlib) [ 9905.564637] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [ 9905.810766] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9907.463236] Key type lgssc registered [ 9908.657668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9915.787466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9915.804390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9917.202136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9923.037910] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9927.649376] LustreError: 321577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9927.677609] LustreError: 321577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9932.801732] LustreError: 321578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9933.647871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9934.126777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9938.693373] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9945.777594] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9950.932864] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9958.960873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9959.452750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.304350] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9973.947744] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9990.630966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9990.641042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9990.656565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9992.505119] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 9994.227125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9994.266919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9997.070714] Lustre: server umount lustre-MDT0000 complete [ 9997.551042] LustreError: 321573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10001.279649] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052510 with bad export cookie 16194172670211811688 [10001.287034] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10001.302314] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10002.674803] LustreError: 322598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.686352] LustreError: 322598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10002.689481] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [10002.693503] Lustre: Skipped 2 previous similar messages [10004.454879] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10004.466311] Lustre: Skipped 1 previous similar message [10007.720543] Lustre: server umount lustre-MDT0001 complete [10015.963557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10024.754279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.887356] LustreError: 325193:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10025.901206] LustreError: 325193:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000002675453b x1875957425154688/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789052535 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10025.929181] LustreError: 325193:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10025.956811] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe0bd423e3f50872f [10025.978548] Lustre: MGC192.168.202.159@tcp: Connection restored to 0@lo (at 0@lo) [10026.359059] LustreError: 323422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10026.377997] LustreError: 323422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10026.444777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10026.512180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10031.990449] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10036.718465] LustreError: 323422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.721054] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10036.738090] LustreError: 323422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10040.606896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10040.961756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10046.181924] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10046.475598] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10053.909793] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10055.714824] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10062.458759] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10064.257468] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10071.521293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10071.527216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10071.539606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10071.549297] Lustre: Skipped 2 previous similar messages [10075.765787] Lustre: server umount lustre-OST0000 complete [10081.760493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10081.771067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10081.783924] Lustre: Skipped 1 previous similar message [10081.789262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10081.796122] Lustre: Skipped 2 previous similar messages [10086.347462] Lustre: server umount lustre-MDT0000 complete [10087.392949] LustreError: 325213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.404368] LustreError: 325213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10090.169220] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052599 with bad export cookie 16194172670211819311 [10090.170947] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10090.176517] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10090.476563] Lustre: server umount lustre-MDT0001 complete [10102.010311] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [10104.672418] Key type lgssc unregistered [10105.051067] LNet: 327879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10105.071994] LNetError: 327879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10105.086611] LNet: Removed LNI 192.168.202.159@tcp [10106.214058] Key type .llcrypt unregistered [10106.217095] Key type ._llcrypt unregistered [10133.310608] Key type ._llcrypt registered [10133.314614] Key type .llcrypt registered [10133.486640] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10134.657597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10134.747717] alg: No test for adler32 (adler32-zlib) [10135.811804] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [10136.030556] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [10137.705165] Key type lgssc registered [10138.960316] Lustre: Echo OBD driver; http://www.lustre.org/ [10146.716057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10146.731289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10148.357698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10152.778414] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10158.580398] LustreError: 329291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10158.592248] LustreError: 329291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10161.111097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.465137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10166.544614] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10174.558467] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10181.019554] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10186.337741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.345023] Lustre: Skipped 1 previous similar message [10186.346382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10186.357816] Lustre: Skipped 1 previous similar message [10190.500912] Lustre: server umount lustre-MDT0000 complete [10191.848194] LustreError: 329291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.872155] LustreError: 329291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10193.890563] LustreError: 329271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052703 with bad export cookie 8719194065417370156 [10193.907159] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10194.151271] Lustre: server umount lustre-MDT0001 complete [10209.772465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.221047] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10210.258493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10210.514730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10215.677412] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10225.532374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10225.718417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10225.770874] Lustre: 331876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10226.109950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10231.201290] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10239.742634] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10247.045352] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10257.129767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10257.258615] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10257.269284] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10257.274870] LustreError: 333765:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10257.279914] LustreError: 333765:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10257.287928] LustreError: 333765:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10257.337890] Lustre: server umount lustre-OST0000 complete [10257.345255] LustreError: 333765:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10269.297581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10269.557401] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10269.757151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10275.494717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10276.031541] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10283.626873] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10293.262238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10293.326752] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10293.417699] Lustre: lustre-OST0001: new disk, initializing [10293.423499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10293.433794] Lustre: 335305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10293.507891] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10299.501248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10299.515448] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10299.633687] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10301.704955] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10308.271662] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10315.239680] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10315.251412] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10315.262696] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10317.624576] Lustre: server umount lustre-OST0001 complete [10320.368275] LustreError: 335573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.392037] LustreError: 335573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10321.888408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10321.897682] LustreError: Skipped 1 previous similar message [10321.905207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.919096] Lustre: Skipped 1 previous similar message [10321.926531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10321.934883] Lustre: Skipped 1 previous similar message [10325.486562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10325.488087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10325.489362] LustreError: 334232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.549470] Lustre: Skipped 1 previous similar message [10327.861253] Lustre: server umount lustre-OST0000 complete [10333.673755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10333.683385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10333.716610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10338.632380] Lustre: server umount lustre-MDT0000 complete [10340.833705] LustreError: 331884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.851343] LustreError: 331884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10342.274169] LustreError: 331871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052851 with bad export cookie 8719194065417370709 [10342.282255] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10342.283894] LustreError: 331871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10342.855742] Lustre: server umount lustre-MDT0001 complete [10355.143992] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [10358.458275] Key type lgssc unregistered [10358.807069] LNet: 337261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10358.818425] LNetError: 337261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10358.838627] LNet: Removed LNI 192.168.202.159@tcp [10359.870132] Key type .llcrypt unregistered [10359.871735] Key type ._llcrypt unregistered [10434.363256] Key type ._llcrypt registered [10434.365317] Key type .llcrypt registered [10434.438513] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [10448.418622] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10449.945313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10449.992430] alg: No test for adler32 (adler32-zlib) [10451.194978] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [10451.429426] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [10453.087192] Key type lgssc registered [10454.347664] Lustre: Echo OBD driver; http://www.lustre.org/ [10489.216845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10489.272678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.526843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10490.582819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10490.641153] Lustre: lustre-MDT0000: new disk, initializing [10490.706598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10490.717315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10496.689754] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10508.406117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10508.549425] Lustre: 342191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10508.616186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10508.621914] Lustre: Skipped 1 previous similar message [10508.737568] Lustre: lustre-MDT0001: new disk, initializing [10508.835509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10508.878686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10508.892157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10513.300221] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10522.492361] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10527.313315] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10533.529492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10533.789818] Lustre: lustre-OST0000: new disk, initializing [10533.792289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10533.796973] Lustre: 344161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10533.859347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10542.232852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10542.254670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10542.521676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10542.738867] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10555.152937] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10559.862555] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10560.079308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10565.260812] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10565.466199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10568.169674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10568.191363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10568.203235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10568.210172] Lustre: Skipped 1 previous similar message [10568.237364] Lustre: Skipped 1 previous similar message [10573.282932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10573.487374] Lustre: server umount lustre-OST0000 complete [10578.403319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.414909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10578.429507] Lustre: Skipped 1 previous similar message [10578.439536] Lustre: Skipped 1 previous similar message [10583.521611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10583.537899] Lustre: Skipped 2 previous similar messages [10584.650128] Lustre: server umount lustre-MDT0000 complete [10590.494046] LustreError: 342184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053099 with bad export cookie 3788174627899505063 [10590.515093] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10591.108525] Lustre: server umount lustre-MDT0001 complete [10602.471584] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:11:49 (1789053109) [10610.450842] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:11:58 (1789053118) [10619.042084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10619.409476] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10619.423094] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10619.453848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10624.484557] LustreError: 346921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10624.873500] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10629.608552] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.033486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10633.473905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10637.578879] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10645.328440] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10652.400787] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10661.054189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10661.854598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10668.754198] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10676.672295] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10686.331919] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10688.691653] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10690.595651] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10692.452264] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10694.696026] Lustre: Failing over lustre-MDT0000 [10694.916193] LustreError: 349350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10694.990851] Lustre: server umount lustre-MDT0000 complete [10695.143900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10695.157443] LustreError: Skipped 1 previous similar message [10695.161928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10698.721692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10698.740615] Lustre: Skipped 1 previous similar message [10700.027210] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.048644] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10708.978978] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10708.997710] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10714.081588] Lustre: 340415:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053207/real 1789053207] req@00000000a6e8653b x1875957996657920/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789053223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10714.092364] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10714.715807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10724.527181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10724.565195] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10726.647669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10729.314231] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10729.974195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10730.032120] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10730.100929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10740.259474] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10742.442915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10746.353545] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10748.547928] Lustre: Failing over lustre-MDT0000 [10748.933419] Lustre: server umount lustre-MDT0000 complete [10750.445240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10750.460239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10750.466741] LustreError: 346921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10750.466753] LustreError: 346921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10750.531449] Lustre: Skipped 1 previous similar message [10765.796762] Lustre: 340416:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053259/real 1789053259] req@000000006f330adf x1875957996678400/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789053275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10765.812435] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10769.019637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10776.535814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10776.622103] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10777.851515] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10781.162126] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10781.684465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10781.692104] Lustre: Skipped 2 previous similar messages [10781.724762] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10781.823463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10791.067627] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10792.759587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10802.145432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.161554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10802.170731] Lustre: Skipped 3 previous similar messages [10802.173383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10802.176243] Lustre: Skipped 2 previous similar messages [10805.282345] Lustre: server umount lustre-MDT0000 complete [10807.268753] LustreError: 347951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.284378] LustreError: 347951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [10809.375148] LustreError: 346900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053318 with bad export cookie 3788174627899507772 [10809.384179] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10809.389071] LustreError: 346900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10809.833455] Lustre: server umount lustre-MDT0001 complete [10823.012932] Lustre: server umount lustre-OST0000 complete [10838.756166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.786783] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10854.678478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10855.238910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10855.258698] Lustre: Skipped 1 previous similar message [10861.245378] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10869.558318] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10875.754473] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10882.993779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10884.571344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10889.786426] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10898.744993] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10914.270517] Lustre: server umount lustre-OST0000 complete [10920.419892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.430418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10920.435094] Lustre: Skipped 1 previous similar message [10920.459709] Lustre: Skipped 1 previous similar message [10928.730276] Lustre: server umount lustre-MDT0000 complete [10930.656992] LustreError: 353258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.671974] LustreError: 353258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10931.169767] Lustre: 340417:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053424/real 1789053424] req@00000000b621cb00 x1875957996733312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789053440 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10932.943132] LustreError: 355107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053442 with bad export cookie 3788174627899508521 [10932.952837] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10932.971083] LustreError: 355107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10933.553756] Lustre: server umount lustre-MDT0001 complete [10950.415518] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [10954.756298] Key type lgssc unregistered [10955.162173] LNet: 356813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10955.183453] LNetError: 356813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10955.215462] LNet: Removed LNI 192.168.202.159@tcp [10956.444222] Key type .llcrypt unregistered [10956.452112] Key type ._llcrypt unregistered [10988.793593] Key type ._llcrypt registered [10988.798729] Key type .llcrypt registered [10988.928861] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10990.083466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10990.281400] alg: No test for adler32 (adler32-zlib) [10991.463930] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [10991.707507] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [10993.391294] Key type lgssc registered [10994.531491] Lustre: Echo OBD driver; http://www.lustre.org/ [11002.728629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11002.763313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.430976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11010.665693] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11014.632653] LustreError: 358328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11014.639748] LustreError: 358328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11018.846287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11019.211422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11023.553214] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11031.369966] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11037.876751] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11045.420949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11045.784364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11048.912569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11053.049766] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11060.097754] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11079.649411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11079.655013] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11079.664916] Lustre: Skipped 2 previous similar messages [11079.667778] Lustre: Skipped 1 previous similar message [11082.898385] Lustre: server umount lustre-MDT0000 complete [11084.769853] LustreError: 358323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11084.795914] LustreError: 358323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11087.434301] LustreError: 358310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053596 with bad export cookie 4092518901530219032 [11087.439276] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11087.454583] LustreError: 358310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11088.364928] Lustre: server umount lustre-MDT0001 complete [11103.192509] Lustre: server umount lustre-OST0000 complete [11117.395249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11117.822408] LustreError: 362062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11117.892339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11122.767657] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11123.170388] LustreError: 362063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11128.307972] LustreError: 362062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.224297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11131.532409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11135.905037] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11145.066253] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11150.159924] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11157.347836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11157.614605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11163.599036] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11163.767661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11172.622564] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11186.658102] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11186.662581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11186.674512] Lustre: Skipped 2 previous similar messages [11186.684986] Lustre: Skipped 3 previous similar messages [11190.282188] Lustre: server umount lustre-MDT0000 complete [11191.777599] LustreError: 362062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.829381] LustreError: 362062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11194.418337] LustreError: 362043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053703 with bad export cookie 4092518901530220145 [11194.434458] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11194.435742] LustreError: 362043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11194.866151] Lustre: server umount lustre-MDT0001 complete [11209.017976] Lustre: server umount lustre-OST0000 complete [11226.541903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11227.005422] LustreError: 365973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11227.082836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11231.526209] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11240.016885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11245.156923] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11252.511868] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11258.065079] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11264.515517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11264.864250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11264.868857] Lustre: Skipped 1 previous similar message [11271.597110] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11272.069608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11278.845489] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11285.960622] Lustre: server umount lustre-OST0000 complete [11291.626353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11291.641734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11291.655210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11297.765455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11297.777073] Lustre: Skipped 2 previous similar messages [11299.949244] Lustre: server umount lustre-MDT0000 complete [11301.792240] Lustre: 357638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053795/real 1789053795] req@000000005293802f x1875958562906368/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789053811 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11301.807682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11301.813721] Lustre: Skipped 1 previous similar message [11301.818373] LustreError: 365973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11301.828147] LustreError: 365973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11302.879443] Lustre: 357640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053796/real 1789053796] req@000000009dba250c x1875958562907264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789053812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11303.126871] LustreError: 365955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053812 with bad export cookie 4092518901530221167 [11303.128172] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11303.144281] LustreError: 365955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11303.321427] Lustre: server umount lustre-MDT0001 complete [11312.201411] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11315.188048] Key type lgssc unregistered [11315.466659] LNet: 369504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11315.488879] LNetError: 369504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11315.542453] LNet: Removed LNI 192.168.202.159@tcp [11316.546746] Key type .llcrypt unregistered [11316.549095] Key type ._llcrypt unregistered [11348.994578] Key type ._llcrypt registered [11348.996966] Key type .llcrypt registered [11349.094876] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11349.788972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11349.860988] alg: No test for adler32 (adler32-zlib) [11351.070884] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [11351.258948] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11352.895230] Key type lgssc registered [11354.140305] Lustre: Echo OBD driver; http://www.lustre.org/ [11360.452555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11360.468647] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.472934] LustreError: 370942:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11360.486383] LustreError: 370942:0:(obd_config.c:920:class_setup()) setup lustre-MDT0000-osd failed (-22) [11360.491604] LustreError: 370942:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11360.498572] LustreError: 370942:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11360.506048] LustreError: 370942:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11367.719867] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11367.737725] LustreError: 371340:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11367.766303] LustreError: 371340:0:(obd_config.c:920:class_setup()) setup lustre-OST0000-osd failed (-22) [11367.777983] LustreError: 371340:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11367.785840] LustreError: 371340:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11367.801363] LustreError: 371340:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11389.415673] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11392.456424] Key type lgssc unregistered [11392.851746] LNet: 372343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11392.860292] LNetError: 372343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11392.876581] LNet: Removed LNI 192.168.202.159@tcp [11393.783366] Key type .llcrypt unregistered [11393.785692] Key type ._llcrypt unregistered [11411.774231] Key type ._llcrypt registered [11411.776962] Key type .llcrypt registered [11411.892847] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [11428.125566] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11429.622055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11429.700484] alg: No test for adler32 (adler32-zlib) [11430.762758] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [11431.079922] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11432.759282] Key type lgssc registered [11434.142806] Lustre: Echo OBD driver; http://www.lustre.org/ [11471.932090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11471.990137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11473.307637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11473.345839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11473.409497] Lustre: lustre-MDT0000: new disk, initializing [11473.625055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11473.661055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11479.207203] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11492.163417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11492.308425] Lustre: 375496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11492.367201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11492.372718] Lustre: Skipped 1 previous similar message [11492.445963] Lustre: lustre-MDT0001: new disk, initializing [11492.588154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11492.674561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11492.706949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11498.053117] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11509.766561] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11516.154470] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11521.570553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11521.792847] Lustre: lustre-OST0000: new disk, initializing [11521.795727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11521.799742] Lustre: 377463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11521.867356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11527.311879] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11528.302722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11528.307260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11528.481689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11537.251912] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11541.100972] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11541.367550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11547.919994] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11548.144693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11554.274045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11554.278064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11554.301392] LustreError: Skipped 1 previous similar message [11554.306398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11554.343448] Lustre: Skipped 1 previous similar message [11554.383186] Lustre: Skipped 1 previous similar message [11556.601632] Lustre: server umount lustre-OST0000 complete [11564.513726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11564.515122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11564.533629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11569.633710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11569.647397] Lustre: Skipped 2 previous similar messages [11574.753780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11574.759096] Lustre: Skipped 1 previous similar message [11576.290063] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11576.601831] Lustre: server umount lustre-MDT0000 complete [11580.774984] LustreError: 375486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054089 with bad export cookie 6250276627689857393 [11580.776309] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11580.796018] LustreError: 375486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11581.261103] Lustre: server umount lustre-MDT0001 complete [11590.212524] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11592.450470] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:28:19 (1789054099) [11599.474173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11599.952334] LustreError: 379925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.968610] LustreError: 379925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11600.072546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11605.074666] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11605.474215] LustreError: 379926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11605.503788] LustreError: 379926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11610.592250] LustreError: 379925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.914990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11613.332680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11617.548444] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11626.494876] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11633.424859] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11640.930918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11641.206569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11647.164425] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11654.902468] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11661.976344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11662.130522] Lustre: lustre-OST0001: new disk, initializing [11662.134649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11662.155899] Lustre: 382767:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11662.294046] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11669.581782] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11671.107285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11671.116780] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11671.245550] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11680.203631] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11686.889489] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11686.894870] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11686.897460] LustreError: Skipped 1 previous similar message [11686.899735] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11686.914899] Lustre: Skipped 2 previous similar messages [11686.930270] Lustre: Skipped 1 previous similar message [11690.325128] Lustre: server umount lustre-OST0001 complete [11692.004613] LustreError: 381747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11692.034408] LustreError: 381747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11697.133451] LustreError: 381749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11697.162846] LustreError: 381749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11707.386402] LustreError: 383203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11707.419850] LustreError: 383203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11709.924828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11709.944223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11709.966445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11709.971731] Lustre: Skipped 1 previous similar message [11714.749464] Lustre: server umount lustre-OST0000 complete [11720.672598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11720.693123] LustreError: Skipped 1 previous similar message [11720.698124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11720.711637] Lustre: Skipped 1 previous similar message [11726.176732] Lustre: server umount lustre-MDT0000 complete [11727.841839] LustreError: 381156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.864112] LustreError: 381156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11730.884539] LustreError: 382765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054240 with bad export cookie 6250276627689858275 [11730.907583] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11737.608070] Lustre: server umount lustre-MDT0001 complete [11750.064501] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11754.747804] Key type lgssc unregistered [11755.376922] LNet: 384755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11755.397581] LNetError: 384755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11756.455505] LNet: Removed LNI 192.168.202.159@tcp [11758.042621] Key type .llcrypt unregistered [11758.045986] Key type ._llcrypt unregistered [11820.340282] Key type ._llcrypt registered [11820.342603] Key type .llcrypt registered [11820.474486] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11822.043206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11822.113575] alg: No test for adler32 (adler32-zlib) [11823.177412] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [11823.397932] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11825.063428] Key type lgssc registered [11826.162400] Lustre: Echo OBD driver; http://www.lustre.org/ [11835.573390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11835.614770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11837.011179] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11837.068640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11837.310140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11843.180693] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11853.268959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.471236] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11853.516185] Lustre: 387553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11853.961756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11859.448400] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11868.468881] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11875.421589] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11879.397760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11879.404422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11879.420840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11879.911309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11879.938252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11884.060741] Lustre: server umount lustre-MDT0000 complete [11888.129767] LustreError: 387546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054397 with bad export cookie 11250481117915726361 [11888.147247] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11888.156123] LustreError: 387546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11888.576302] Lustre: server umount lustre-MDT0001 complete [11898.869641] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:33:26 (1789054406) [11906.325715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11906.752689] LustreError: 390067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11906.769241] LustreError: 390067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11906.851484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11911.097174] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11912.168761] LustreError: 390068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.291700] LustreError: 390067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11920.827633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.169634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11926.748328] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11935.093863] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11941.744287] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11949.042447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11949.471847] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11949.746279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11956.777289] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11963.445541] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11971.627981] Lustre: Permanently deactivating lustre-OST0000 [11971.668398] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11979.929341] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11982.299955] LustreError: 393052:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11982.317912] LustreError: 393052:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11982.327731] LustreError: 393052:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11986.645789] LustreError: 393100:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11986.661313] LustreError: 393100:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11995.208498] Lustre: server umount lustre-OST0000 complete [11998.187961] LustreError: 393351:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11998.201785] LustreError: 393351:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12001.250559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12001.266224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12001.281604] Lustre: Skipped 1 previous similar message [12001.305030] Lustre: Skipped 1 previous similar message [12006.368908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12010.533047] Lustre: server umount lustre-MDT0000 complete [12011.490938] LustreError: 390063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12011.511674] LustreError: 390063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12013.913842] LustreError: 390798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054523 with bad export cookie 11250481117915726942 [12013.920622] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12013.926272] LustreError: 390798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12024.575499] Lustre: server umount lustre-MDT0001 complete [12034.871980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12041.462629] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12044.696542] LustreError: 394511:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12050.548101] Lustre: 394851:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.159@tcp [12053.058305] Lustre: server umount MGS complete [12066.474756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12066.745029] LustreError: 395575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12066.751231] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12066.756285] Lustre: Skipped 1 previous similar message [12066.785798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12073.136428] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12077.027869] LustreError: 395576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12077.040361] LustreError: 395576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12082.960725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12083.511097] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12083.679205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12088.874363] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12096.957911] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12102.885709] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12105.035397] Lustre: Permanently reactivating lustre-OST0000 [12105.045058] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12105.047015] Lustre: Skipped 2 previous similar messages [12135.391157] LustreError: 397405:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12144.342436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12144.758177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12152.005908] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12159.670652] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12165.613067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12165.637213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12165.650136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12165.665236] Lustre: Skipped 2 previous similar messages [12166.402203] Lustre: server umount lustre-OST0000 complete [12170.723550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12170.732085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12170.756279] Lustre: Skipped 2 previous similar messages [12175.840669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12175.843109] Lustre: Skipped 3 previous similar messages [12176.447041] Lustre: server umount lustre-MDT0000 complete [12180.418197] LustreError: 395557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054689 with bad export cookie 11250481117915727985 [12180.419983] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12180.429880] LustreError: 395557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12180.778323] Lustre: server umount lustre-MDT0001 complete [12196.181943] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12199.208306] Key type lgssc unregistered [12199.573984] LNet: 400042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12199.584573] LNetError: 400042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12199.606403] LNet: Removed LNI 192.168.202.159@tcp [12200.860757] Key type .llcrypt unregistered [12200.862062] Key type ._llcrypt unregistered [12223.628256] Key type ._llcrypt registered [12223.631789] Key type .llcrypt registered [12223.787078] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [12241.947672] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12243.191431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12243.246599] alg: No test for adler32 (adler32-zlib) [12244.270958] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [12244.434158] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12246.111413] Key type lgssc registered [12247.678754] Lustre: Echo OBD driver; http://www.lustre.org/ [12285.522558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12285.538486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12286.752258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12286.781605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12286.929962] Lustre: lustre-MDT0000: new disk, initializing [12287.107867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12287.149366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12291.668928] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12304.756902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12304.998256] Lustre: 403196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12305.081480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12305.088483] Lustre: Skipped 1 previous similar message [12305.192629] Lustre: lustre-MDT0001: new disk, initializing [12305.348329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12305.408312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12305.421653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12312.139815] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12323.815306] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12329.597341] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12336.545570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12336.812392] Lustre: lustre-OST0000: new disk, initializing [12336.817460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12336.825913] Lustre: 405167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12336.915860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12343.794192] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12345.429094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12345.460633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12345.648108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12354.946945] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12361.135217] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12361.490822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12367.045174] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12367.327402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12371.434824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12371.434834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12371.446081] LustreError: Skipped 1 previous similar message [12371.450150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12371.483358] Lustre: Skipped 1 previous similar message [12376.548520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12376.555180] Lustre: Skipped 1 previous similar message [12381.674883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12381.694383] Lustre: Skipped 1 previous similar message [12384.225042] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12384.579579] Lustre: server umount lustre-OST0000 complete [12391.908491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12391.948148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12391.951395] Lustre: Skipped 1 previous similar message [12397.032665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12397.041742] Lustre: Skipped 1 previous similar message [12403.170674] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12403.600701] Lustre: server umount lustre-MDT0000 complete [12408.669489] LustreError: 405166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054917 with bad export cookie 8524740483654518566 [12408.674517] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12408.690027] LustreError: 405166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12409.370447] Lustre: server umount lustre-MDT0001 complete [12419.521225] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12421.572714] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:42:08 (1789054928) [12429.443203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12429.921345] LustreError: 407630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12429.936495] LustreError: 407630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12430.041323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12435.053334] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12435.439274] LustreError: 407631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12435.472040] LustreError: 407631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12440.553217] LustreError: 407630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12444.823246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12445.308128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12450.109667] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12457.449431] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12463.351425] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12470.803758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12471.322758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12479.721205] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12490.497662] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12492.892149] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12495.890616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12506.089110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12506.092531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12506.094906] LustreError: Skipped 1 previous similar message [12506.104338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12506.104351] Lustre: Skipped 2 previous similar messages [12506.139336] Lustre: Skipped 2 previous similar messages [12508.584916] Lustre: server umount lustre-OST0000 complete [12516.325165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12519.082572] Lustre: server umount lustre-MDT0000 complete [12521.443544] LustreError: 407630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12521.461631] LustreError: 407630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12523.516265] LustreError: 407611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055032 with bad export cookie 8524740483654519448 [12523.519494] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12523.530407] LustreError: 407611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12524.166970] Lustre: server umount lustre-MDT0001 complete [12538.127638] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12541.095538] Key type lgssc unregistered [12541.379984] LNet: 411356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12541.387121] LNetError: 411356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12541.400658] LNet: Removed LNI 192.168.202.159@tcp [12542.683623] Key type .llcrypt unregistered [12542.686234] Key type ._llcrypt unregistered [12568.442339] Key type ._llcrypt registered [12568.444431] Key type .llcrypt registered [12568.590577] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12568.864718] Key type .llcrypt unregistered [12568.869170] Key type ._llcrypt unregistered [12587.796583] Key type ._llcrypt registered [12587.800055] Key type .llcrypt registered [12587.897093] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12589.348493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12589.390328] alg: No test for adler32 (adler32-zlib) [12590.753768] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [12591.264839] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12593.007630] Key type lgssc registered [12594.834532] Lustre: Echo OBD driver; http://www.lustre.org/ [12607.922700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12607.992827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12610.339476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12616.608929] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12620.779616] LustreError: 413563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.787489] LustreError: 413563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12625.888214] LustreError: 413564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12626.406970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12626.691779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12632.048694] LustreError: 413559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12632.095717] LustreError: 413559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12634.001167] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12636.207739] LustreError: 413564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12636.236792] LustreError: 413564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12636.267154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12641.254901] LustreError: 413563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12641.278530] LustreError: 413563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12642.659029] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12650.000679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12650.507024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12655.058589] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12665.831077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12665.837830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12665.863932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12670.909931] Lustre: server umount lustre-OST0000 complete [12676.066063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12676.081769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12676.085190] Lustre: Skipped 2 previous similar messages [12676.088195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12676.088203] Lustre: Skipped 1 previous similar message [12681.186678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12681.189400] Lustre: Skipped 1 previous similar message [12682.214703] Lustre: server umount lustre-MDT0000 complete [12686.305564] LustreError: 415299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.323054] LustreError: 415299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12686.948039] LustreError: 413544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055196 with bad export cookie 13122665211675938372 [12686.965414] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12686.966453] LustreError: 413544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12687.445203] Lustre: server umount lustre-MDT0001 complete [12698.871653] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12701.173458] Key type lgssc unregistered [12701.480823] LNet: 416711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12701.489112] LNetError: 416711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12701.509640] LNet: Removed LNI 192.168.202.159@tcp [12702.490612] Key type .llcrypt unregistered [12702.493280] Key type ._llcrypt unregistered [12733.757477] Key type ._llcrypt registered [12733.758844] Key type .llcrypt registered [12733.969202] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12736.082352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12736.151448] alg: No test for adler32 (adler32-zlib) [12737.484945] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [12737.887822] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12739.671204] Key type lgssc registered [12741.207191] Lustre: Echo OBD driver; http://www.lustre.org/ [12750.615731] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12750.653922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12766.175388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12771.296477] LustreError: 418097:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [12798.047896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12805.515545] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12813.738104] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12821.283537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12822.323991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12822.469294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12827.607365] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12827.618834] LustreError: 418123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.637902] LustreError: 418123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12832.740564] LustreError: 418122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12833.774292] LustreError: 419135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12833.794159] LustreError: 419135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12836.920018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12837.339500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12843.015267] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12851.478371] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12858.255049] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12867.571935] Lustre: server umount lustre-OST0000 complete [12873.185400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12873.189330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12873.196989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12873.199791] LustreError: Skipped 1 previous similar message [12873.224989] Lustre: Skipped 1 previous similar message [12878.307297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12878.320587] Lustre: Skipped 2 previous similar messages [12882.510255] Lustre: server umount lustre-MDT0000 complete [12883.430946] LustreError: 419141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12883.457804] LustreError: 419141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12884.452289] Lustre: 417433:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055377/real 1789055377] req@0000000013351ea2 x1875960394709120/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789055393 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12884.458348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12884.484876] Lustre: 417433:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12886.551687] LustreError: 418104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055395 with bad export cookie 6344900137374834181 [12886.562519] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12886.566819] LustreError: 418104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12887.194712] Lustre: server umount lustre-MDT0001 complete [12899.248561] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12903.200920] Key type lgssc unregistered [12903.543666] LNet: 421668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12903.566035] LNetError: 421668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12903.588534] LNet: Removed LNI 192.168.202.159@tcp [12904.600325] Key type .llcrypt unregistered [12904.604469] Key type ._llcrypt unregistered [12936.156150] Key type ._llcrypt registered [12936.164108] Key type .llcrypt registered [12936.297353] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12938.125926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12938.154436] alg: No test for adler32 (adler32-zlib) [12939.436339] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [12939.780668] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12941.503190] Key type lgssc registered [12942.919305] Lustre: Echo OBD driver; http://www.lustre.org/ [12953.821926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12953.838064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12955.533511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12961.845817] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12964.840138] LustreError: 423087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12964.861859] LustreError: 423087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12969.952850] LustreError: 423088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12973.319635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12973.849754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12980.518442] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12989.112254] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12996.728751] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13004.729361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13005.097168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13012.557838] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13024.022658] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13025.614976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13035.270436] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [13035.494954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13035.511379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13036.005279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13036.016440] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13036.023793] Lustre: Skipped 1 previous similar message [13036.061405] Lustre: Skipped 1 previous similar message [13038.727977] Lustre: server umount lustre-MDT0000 complete [13040.370245] LustreError: 424184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13041.506451] LustreError: 423069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.59@tcp arrived at 1789055550 with bad export cookie 12085256008133939844 [13043.378894] LustreError: 423068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055552 with bad export cookie 12085256008133939179 [13043.391892] LustreError: 423068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13043.394280] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13044.193878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13044.202070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13044.225364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13044.252046] Lustre: Skipped 1 previous similar message [13046.241862] LustreError: 423088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13046.245418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13046.266143] LustreError: 423088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13049.750723] Lustre: server umount lustre-OST0000 complete [13053.413344] LustreError: 423082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13053.433282] LustreError: 423082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13058.053382] Lustre: server umount lustre-MDT0001 complete [13070.922210] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13074.299683] Key type lgssc unregistered [13074.675815] LNet: 426774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13074.685987] LNetError: 426774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13074.699902] LNet: Removed LNI 192.168.202.159@tcp [13075.558966] Key type .llcrypt unregistered [13075.561968] Key type ._llcrypt unregistered [13103.758954] Key type ._llcrypt registered [13103.760956] Key type .llcrypt registered [13103.855622] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [13105.015605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13105.096777] alg: No test for adler32 (adler32-zlib) [13106.097483] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [13106.376129] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [13108.151186] Key type lgssc registered [13109.380750] Lustre: Echo OBD driver; http://www.lustre.org/ [13118.448839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13118.492506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13120.077706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13126.072064] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13130.215509] LustreError: 428187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13130.236056] LustreError: 428187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13135.334843] LustreError: 428188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13137.282852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13137.798687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13143.730787] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13151.934748] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13158.121386] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13165.998769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13166.602347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13173.303312] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13181.150711] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13190.150174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13198.613054] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [13199.350743] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13199.362059] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13199.382348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13200.355405] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13202.615367] Lustre: server umount lustre-MDT0001 complete [13203.698456] LustreError: 428184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.592527] LustreError: 428188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.607749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13210.632162] LustreError: 428188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13210.656346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13210.667700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13210.678633] Lustre: Skipped 2 previous similar messages [13215.714823] LustreError: 428916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13215.739617] Lustre: server umount lustre-OST0000 complete [13219.621664] Lustre: server umount lustre-MDT0000 complete [13235.573625] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13241.637519] Key type lgssc unregistered [13242.134411] LNet: 431897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13242.159128] LNetError: 431897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13242.182584] LNet: Removed LNI 192.168.202.159@tcp [13243.305548] Key type .llcrypt unregistered [13243.308478] Key type ._llcrypt unregistered [13278.150402] Key type ._llcrypt registered [13278.153041] Key type .llcrypt registered [13278.250902] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [13295.478294] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [13296.654580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13296.687470] alg: No test for adler32 (adler32-zlib) [13297.834456] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [13298.185511] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [13299.895945] Key type lgssc registered [13301.438397] Lustre: Echo OBD driver; http://www.lustre.org/ [13343.621742] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [13364.957932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13365.056458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13366.519402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13366.553358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13366.646866] Lustre: lustre-MDT0000: new disk, initializing [13366.713716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13366.729370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13372.337304] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13396.622816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13397.034254] Lustre: lustre-OST0000: new disk, initializing [13397.040443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13397.044302] Lustre: Skipped 1 previous similar message [13397.049618] Lustre: 436931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13397.212711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13402.706268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13402.724434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13402.743980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13405.697730] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13417.328395] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13437.989229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13438.137509] Lustre: 435913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13438.211113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13438.317575] Lustre: lustre-MDT0001: new disk, initializing [13438.511957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13438.578682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13438.603120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13443.913179] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13454.653233] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13454.918841] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13460.711288] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13460.979665] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13470.688986] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13470.693288] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13470.713616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13470.730691] Lustre: Skipped 1 previous similar message [13472.589812] Lustre: server umount lustre-MDT0001 complete [13480.940262] LustreError: 435918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13480.966925] LustreError: 435918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13486.054872] LustreError: 438024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13486.076784] LustreError: 438024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13491.174288] LustreError: 435924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13491.179115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13491.212522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13491.222174] Lustre: Skipped 2 previous similar messages [13496.289587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13496.296430] LustreError: 435923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13497.368068] Lustre: server umount lustre-OST0000 complete [13509.974935] Lustre: server umount lustre-MDT0000 complete [13530.295837] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13534.042770] Key type lgssc unregistered [13534.456743] LNet: 440681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13534.466590] LNetError: 440681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13534.504649] LNet: Removed LNI 192.168.202.159@tcp [13535.636687] Key type .llcrypt unregistered [13535.638983] Key type ._llcrypt unregistered [13582.883558] Key type ._llcrypt registered [13582.885113] Key type .llcrypt registered [13583.005746] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [13583.460285] Key type .llcrypt unregistered [13583.468134] Key type ._llcrypt unregistered [13602.543193] Key type ._llcrypt registered [13602.545175] Key type .llcrypt registered [13602.634706] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [13604.246530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13604.285653] alg: No test for adler32 (adler32-zlib) [13605.391249] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [13605.734202] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [13607.471732] Key type lgssc registered [13608.769925] Lustre: Echo OBD driver; http://www.lustre.org/ [13639.960552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13640.020145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13641.392567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13641.429642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13641.570670] Lustre: lustre-MDT0000: new disk, initializing [13641.682114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13641.704361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13646.563158] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13658.370218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13658.555487] Lustre: 444066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13658.680073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13658.696589] Lustre: Skipped 1 previous similar message [13658.921047] Lustre: lustre-MDT0001: new disk, initializing [13659.090054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13659.163279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13659.182294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13663.123597] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13676.256428] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13684.102652] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13691.257318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13691.579407] Lustre: lustre-OST0000: new disk, initializing [13691.581929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13691.585819] Lustre: 446032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13691.655501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13697.193435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13697.210433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13697.465102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13700.135726] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13711.233765] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13720.544330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13720.555649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13720.576943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13723.107283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13723.110066] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13723.115577] Lustre: Skipped 1 previous similar message [13723.139071] Lustre: Skipped 1 previous similar message [13728.021362] Lustre: server umount lustre-MDT0000 complete [13731.714695] LustreError: 444057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056240 with bad export cookie 10260175656656043143 [13731.735825] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13732.092964] Lustre: server umount lustre-MDT0001 complete [13746.333864] Lustre: server umount lustre-OST0000 complete [13759.399621] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:04:26 (1789056266) [13768.694344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13769.219633] LustreError: 448062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13769.245416] LustreError: 448062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13769.264493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13773.603768] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13774.312774] LustreError: 448063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13779.427148] LustreError: 448062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13782.630898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13782.864302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13787.823736] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13795.823901] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13804.633438] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13828.598700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13828.965041] Lustre: Found index 0 for lustre-OST0000, updating log [13829.187831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13835.449944] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13840.463462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13845.379689] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13853.602614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13853.629367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13853.635433] Lustre: Skipped 1 previous similar message [13857.978318] Lustre: server umount lustre-OST0000 complete [13875.168331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13875.178558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13875.201545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13875.208393] Lustre: Skipped 2 previous similar messages [13880.873474] Lustre: server umount lustre-MDT0000 complete [13881.322481] LustreError: 448062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13881.338771] LustreError: 448062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13884.886829] LustreError: 448043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056394 with bad export cookie 10260175656656044536 [13884.891928] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13884.894088] LustreError: 448043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13885.230724] Lustre: server umount lustre-MDT0001 complete [13894.560599] LNet: 1 local NIs in recovery (showing 1): 192.168.202.159@tcp [13894.596493] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:06:42 (1789056402) [14527.554904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14528.371045] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14528.396250] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14528.448069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14533.603459] LustreError: 454479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14533.699965] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14538.721476] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14543.271185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14543.773788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14549.661892] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14559.210730] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14566.409849] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14585.760375] LNet: 1 local NIs in recovery (showing 1): 192.168.202.159@tcp [14587.166815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14587.948286] Lustre: Found index 0 for lustre-OST0000, updating log [14588.679751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14597.327485] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14608.976173] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14612.449399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14612.453751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14612.465454] Lustre: Skipped 1 previous similar message [14612.475050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14612.485722] Lustre: Skipped 2 previous similar messages [14618.546605] Lustre: server umount lustre-OST0000 complete [14623.811187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14623.817443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14623.828877] Lustre: Skipped 1 previous similar message [14623.839539] Lustre: Skipped 1 previous similar message [14624.867318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14624.874472] Lustre: Skipped 1 previous similar message [14626.919083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14626.937970] Lustre: Skipped 3 previous similar messages [14630.054524] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14630.091254] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14630.498591] Lustre: server umount lustre-MDT0000 complete [14634.150863] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14634.168691] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [14634.799797] LustreError: 454458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057144 with bad export cookie 10260175656656045607 [14634.806167] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14634.812131] LustreError: 454458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14635.555945] Lustre: server umount lustre-MDT0001 complete [14645.216276] LNet: 1 local NIs in recovery (showing 1): 192.168.202.159@tcp [14670.843415] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [14684.821706] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [14730.680773] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:20:37 (1789057237) [14753.484075] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [14758.965977] Key type lgssc unregistered [14759.332779] LNet: 461101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14759.347328] LNetError: 461101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14849.510502] LNet: Removed LNI 192.168.202.159@tcp [14850.998205] Key type .llcrypt unregistered [14851.000890] Key type ._llcrypt unregistered [14871.561616] Key type ._llcrypt registered [14871.567768] Key type .llcrypt registered [14871.735305] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [14873.212677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14873.286053] alg: No test for adler32 (adler32-zlib) [14874.474069] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [14874.765710] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [14876.544564] Key type lgssc registered [14877.891223] Lustre: Echo OBD driver; http://www.lustre.org/ [14883.068602] LNet: Added LNI 192.168.202.159@tcp100 [8/256/0/180] [14884.319134] LNet: Added LNI 192.168.202.159@tcp101 [8/256/0/180] [14885.548261] LNet: Added LNI 192.168.202.159@tcp102 [8/256/0/180] [14886.594924] LNet: Added LNI 192.168.202.159@tcp103 [8/256/0/180] [14887.714295] LNet: Added LNI 192.168.202.159@tcp104 [8/256/0/180] [14888.695595] LNet: Added LNI 192.168.202.159@tcp105 [8/256/0/180] [14889.724631] LNet: Added LNI 192.168.202.159@tcp106 [8/256/0/180] [14890.693090] LNet: Added LNI 192.168.202.159@tcp107 [8/256/0/180] [14891.792786] LNet: Added LNI 192.168.202.159@tcp108 [8/256/0/180] [14892.784505] LNet: Added LNI 192.168.202.159@tcp109 [8/256/0/180] [14893.789841] LNet: Added LNI 192.168.202.159@tcp110 [8/256/0/180] [14894.707433] LNet: Added LNI 192.168.202.159@tcp111 [8/256/0/180] [14895.690283] LNet: Added LNI 192.168.202.159@tcp112 [8/256/0/180] [14896.805777] LNet: Added LNI 192.168.202.159@tcp113 [8/256/0/180] [14897.867685] LNet: Added LNI 192.168.202.159@tcp114 [8/256/0/180] [14898.759913] LNet: Added LNI 192.168.202.159@tcp115 [8/256/0/180] [14899.821587] LNet: Added LNI 192.168.202.159@tcp116 [8/256/0/180] [14900.805879] LNet: Added LNI 192.168.202.159@tcp117 [8/256/0/180] [14901.728100] LNet: Added LNI 192.168.202.159@tcp118 [8/256/0/180] [14902.801743] LNet: Added LNI 192.168.202.159@tcp119 [8/256/0/180] [14903.845105] LNet: Added LNI 192.168.202.159@tcp120 [8/256/0/180] [14904.906272] LNet: Added LNI 192.168.202.159@tcp121 [8/256/0/180] [14906.043627] LNet: Added LNI 192.168.202.159@tcp122 [8/256/0/180] [14907.126051] LNet: Added LNI 192.168.202.159@tcp123 [8/256/0/180] [14908.085450] LNet: Added LNI 192.168.202.159@tcp124 [8/256/0/180] [14909.077739] LNet: Added LNI 192.168.202.159@tcp125 [8/256/0/180] [14910.150835] LNet: Added LNI 192.168.202.159@tcp126 [8/256/0/180] [14911.171542] LNet: Added LNI 192.168.202.159@tcp127 [8/256/0/180] [14912.159412] LNet: Added LNI 192.168.202.159@tcp128 [8/256/0/180] [14913.072341] LNet: Added LNI 192.168.202.159@tcp129 [8/256/0/180] [14914.141546] LNet: Added LNI 192.168.202.159@tcp130 [8/256/0/180] [14915.096489] LNet: Added LNI 192.168.202.159@tcp131 [8/256/0/180] [14915.973525] LNet: Added LNI 192.168.202.159@tcp132 [8/256/0/180] [14916.975952] LNet: Added LNI 192.168.202.159@tcp133 [8/256/0/180] [14917.984985] LNet: Added LNI 192.168.202.159@tcp134 [8/256/0/180] [14919.321582] LNet: Added LNI 192.168.202.159@tcp135 [8/256/0/180] [14927.323246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14927.355844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14928.572360] Lustre: 464125:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.159@tcp: can't add NID 192.168.202.159@tcp130: rc = -75 [14928.704943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14928.725374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14928.784339] Lustre: lustre-MDT0000: new disk, initializing [14928.886729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14928.905545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14934.047310] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14944.757131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14945.017482] Lustre: 464142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14945.078307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14945.081553] Lustre: Skipped 1 previous similar message [14945.155854] Lustre: lustre-MDT0001: new disk, initializing [14945.198926] Lustre: 464939:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.159@tcp: can't add NID 192.168.202.159@tcp130: rc = -75 [14945.221026] Lustre: 464939:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14945.450602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14945.513487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14945.536636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14951.685229] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14965.850616] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14974.639788] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14982.826549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14983.346826] Lustre: lustre-OST0000: new disk, initializing [14983.350197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14983.354639] Lustre: 466113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14983.533340] Lustre: 466118:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.159@tcp: can't add NID 192.168.202.159@tcp130: rc = -75 [14983.541134] Lustre: 466118:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14983.606173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14991.528880] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14993.449300] Lustre: 466543:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.159@tcp: can't add NID 192.168.202.159@tcp130: rc = -75 [14993.453646] Lustre: 466543:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14993.556915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14993.570665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14993.753424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15002.809548] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15014.378083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15014.381666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15014.388846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15014.424894] Lustre: Skipped 1 previous similar message [15018.642118] Lustre: server umount lustre-OST0000 complete [15027.171410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15027.183667] LustreError: Skipped 1 previous similar message [15027.190439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15027.227395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15027.236634] Lustre: Skipped 1 previous similar message [15029.730266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15029.732319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15029.765876] Lustre: Skipped 1 previous similar message [15031.760542] Lustre: server umount lustre-MDT0000 complete [15037.353132] LustreError: 467269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057546 with bad export cookie 2409542667434124195 [15037.355253] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15037.362381] LustreError: 467269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15037.726319] Lustre: server umount lustre-MDT0001 complete [15051.618648] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [15054.798017] Key type lgssc unregistered [15055.232896] LNet: 468029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15055.239901] LNetError: 468029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15055.247699] LNet: Removed LNI 192.168.202.159@tcp [15056.287348] LNet: Removed LNI 192.168.202.159@tcp100 [15056.293911] LNetError: 468029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15056.316319] LNetError: 468029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15056.329474] LNet: Removed LNI 192.168.202.159@tcp101 [15057.376417] LNet: Removed LNI 192.168.202.159@tcp102 [15057.378980] LNetError: 468029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15057.381907] LNetError: 468029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15058.399488] LNet: Removed LNI 192.168.202.159@tcp103 [15058.403284] LNet: Removed LNI 192.168.202.159@tcp104 [15058.406194] LNet: Removed LNI 192.168.202.159@tcp105 [15058.409494] LNet: Removed LNI 192.168.202.159@tcp106 [15058.412516] LNet: Removed LNI 192.168.202.159@tcp107 [15058.416224] LNet: Removed LNI 192.168.202.159@tcp108 [15058.420789] LNet: Removed LNI 192.168.202.159@tcp109 [15058.423693] LNet: Removed LNI 192.168.202.159@tcp110 [15058.425472] LNet: Removed LNI 192.168.202.159@tcp111 [15058.427399] LNet: Removed LNI 192.168.202.159@tcp112 [15058.429583] LNet: Removed LNI 192.168.202.159@tcp113 [15058.432509] LNet: Removed LNI 192.168.202.159@tcp114 [15058.435020] LNet: Removed LNI 192.168.202.159@tcp115 [15058.436794] LNet: Removed LNI 192.168.202.159@tcp116 [15058.439073] LNet: Removed LNI 192.168.202.159@tcp117 [15058.440946] LNet: Removed LNI 192.168.202.159@tcp118 [15058.443529] LNet: Removed LNI 192.168.202.159@tcp119 [15058.445453] LNet: Removed LNI 192.168.202.159@tcp120 [15058.448267] LNet: Removed LNI 192.168.202.159@tcp121 [15058.450887] LNet: Removed LNI 192.168.202.159@tcp122 [15058.452754] LNet: Removed LNI 192.168.202.159@tcp123 [15058.455568] LNet: Removed LNI 192.168.202.159@tcp124 [15058.458514] LNet: Removed LNI 192.168.202.159@tcp125 [15058.461561] LNet: Removed LNI 192.168.202.159@tcp126 [15058.464888] LNet: Removed LNI 192.168.202.159@tcp127 [15058.467809] LNet: Removed LNI 192.168.202.159@tcp128 [15058.469887] LNet: Removed LNI 192.168.202.159@tcp129 [15058.472514] LNet: Removed LNI 192.168.202.159@tcp130 [15058.475807] LNet: Removed LNI 192.168.202.159@tcp131 [15058.478517] LNet: Removed LNI 192.168.202.159@tcp132 [15058.481772] LNet: Removed LNI 192.168.202.159@tcp133 [15058.483366] LNet: Removed LNI 192.168.202.159@tcp134 [15058.495132] LNet: Removed LNI 192.168.202.159@tcp135 [15059.343800] Key type .llcrypt unregistered [15059.346465] Key type ._llcrypt unregistered [15088.468475] Key type ._llcrypt registered [15088.470933] Key type .llcrypt registered [15088.637985] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [15107.080972] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [15108.964803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15109.048321] alg: No test for adler32 (adler32-zlib) [15110.151327] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [15110.494984] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [15112.273990] Key type lgssc registered [15113.753569] Lustre: Echo OBD driver; http://www.lustre.org/ [15155.476504] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:27:42 (1789057662) [15175.210450] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [15178.595453] Key type lgssc unregistered [15179.075861] LNet: 472114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15179.082656] LNetError: 472114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15179.102781] LNet: Removed LNI 192.168.202.159@tcp [15179.770742] Key type .llcrypt unregistered [15179.772549] Key type ._llcrypt unregistered [15198.459889] Key type ._llcrypt registered [15198.462609] Key type .llcrypt registered [15198.571694] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [15199.547722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15199.742087] alg: No test for adler32 (adler32-zlib) [15200.940493] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [15201.197432] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [15202.864867] Key type lgssc registered [15204.059202] Lustre: Echo OBD driver; http://www.lustre.org/ [15214.450384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15214.501327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15215.893691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15215.960467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15216.074528] Lustre: lustre-MDT0000: new disk, initializing [15216.173975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15216.206163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15221.877587] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [15234.313250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15234.555818] Lustre: 473322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15234.638226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15234.658198] Lustre: Skipped 1 previous similar message [15234.855534] Lustre: lustre-MDT0001: new disk, initializing [15234.991462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15235.055800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15235.077485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15240.606160] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [15250.999405] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15258.669648] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15266.043350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15266.445628] Lustre: lustre-OST0000: new disk, initializing [15266.450551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15266.457753] Lustre: 475294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15266.566073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15274.354982] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [15275.632681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15275.658212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15275.893197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15284.264371] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15288.924476] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing lnet_if_list [15294.450340] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15294.510666] LNet: Added LNI 192.168.202.159@tcp10 [8/256/0/180] [15302.562931] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15302.617518] LNet: Added LNI 192.168.202.159@tcp11 [8/256/0/180] [15307.216747] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15307.247194] LNet: Added LNI 192.168.202.159@tcp12 [8/256/0/180] [15312.926704] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15312.979695] LNet: Added LNI 192.168.202.159@tcp13 [8/256/0/180] [15319.253264] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15319.282948] LNet: Added LNI 192.168.202.159@tcp14 [8/256/0/180] [15325.432439] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15325.484741] LNet: Added LNI 192.168.202.159@tcp15 [8/256/0/180] [15331.712518] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15331.745310] LNet: Added LNI 192.168.202.159@tcp16 [8/256/0/180] [15338.102445] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15338.166113] LNet: Added LNI 192.168.202.159@tcp17 [8/256/0/180] [15345.016474] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15345.082288] LNet: Added LNI 192.168.202.159@tcp18 [8/256/0/180] [15351.787975] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15351.862216] LNet: Added LNI 192.168.202.159@tcp19 [8/256/0/180] [15357.637574] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15357.659338] LNet: Added LNI 192.168.202.159@tcp20 [8/256/0/180] [15362.865821] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15362.893695] LNet: Added LNI 192.168.202.159@tcp21 [8/256/0/180] [15369.681539] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15369.727950] LNet: Added LNI 192.168.202.159@tcp22 [8/256/0/180] [15377.775892] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15377.820508] LNet: Added LNI 192.168.202.159@tcp23 [8/256/0/180] [15384.438294] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15384.483377] LNet: Added LNI 192.168.202.159@tcp24 [8/256/0/180] [15390.580839] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15390.598268] LNet: Added LNI 192.168.202.159@tcp25 [8/256/0/180] [15395.447556] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15395.516794] LNet: Added LNI 192.168.202.159@tcp26 [8/256/0/180] [15403.363643] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15403.392390] LNet: Added LNI 192.168.202.159@tcp27 [8/256/0/180] [15411.849748] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15411.883880] LNet: Added LNI 192.168.202.159@tcp28 [8/256/0/180] [15421.570876] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15421.609740] LNet: Added LNI 192.168.202.159@tcp29 [8/256/0/180] [15430.339810] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15430.371439] LNet: Added LNI 192.168.202.159@tcp30 [8/256/0/180] [15438.980315] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15439.002535] LNet: Added LNI 192.168.202.159@tcp31 [8/256/0/180] [15446.413070] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15446.454245] LNet: Added LNI 192.168.202.159@tcp32 [8/256/0/180] [15454.661590] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15454.751578] LNet: Added LNI 192.168.202.159@tcp33 [8/256/0/180] [15463.175873] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15463.196606] LNet: Added LNI 192.168.202.159@tcp34 [8/256/0/180] [15470.953679] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15470.979995] LNet: Added LNI 192.168.202.159@tcp35 [8/256/0/180] [15479.946397] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15479.996414] LNet: Added LNI 192.168.202.159@tcp36 [8/256/0/180] [15488.671396] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15488.690504] LNet: Added LNI 192.168.202.159@tcp37 [8/256/0/180] [15499.777357] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15499.804456] LNet: Added LNI 192.168.202.159@tcp38 [8/256/0/180] [15509.359592] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15509.392955] LNet: Added LNI 192.168.202.159@tcp39 [8/256/0/180] [15519.449275] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15519.470511] LNet: Added LNI 192.168.202.159@tcp40 [8/256/0/180] [15528.964503] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15528.992672] LNet: Added LNI 192.168.202.159@tcp41 [8/256/0/180] [15538.743200] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15538.801507] LNet: Added LNI 192.168.202.159@tcp42 [8/256/0/180] [15548.011885] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15548.038457] LNet: Added LNI 192.168.202.159@tcp43 [8/256/0/180] [15557.306757] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15557.343765] LNet: Added LNI 192.168.202.159@tcp44 [8/256/0/180] [15566.497387] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15566.534478] LNet: Added LNI 192.168.202.159@tcp45 [8/256/0/180] [15577.919654] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15577.939710] LNet: Added LNI 192.168.202.159@tcp46 [8/256/0/180] [15586.586597] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15586.626593] LNet: Added LNI 192.168.202.159@tcp47 [8/256/0/180] [15598.378988] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15598.463837] LNet: Added LNI 192.168.202.159@tcp48 [8/256/0/180] [15615.202599] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15615.279249] LNet: Added LNI 192.168.202.159@tcp49 [8/256/0/180] [15625.602063] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15625.653527] LNet: Added LNI 192.168.202.159@tcp50 [8/256/0/180] [15649.761969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15649.766489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15649.805650] Lustre: Skipped 1 previous similar message [15649.820692] Lustre: Skipped 1 previous similar message [15655.070180] Lustre: server umount lustre-MDT0000 complete [15659.213041] LustreError: 474118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058168 with bad export cookie 4350244390571667794 [15659.230103] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15659.237508] LustreError: 474118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15660.077352] Lustre: server umount lustre-MDT0001 complete [15676.383188] Lustre: 472595:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789058169/real 1789058169] req@00000000318362e8 x1875962977514624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789058185 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15676.405653] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15676.414807] Lustre: Skipped 1 previous similar message [15680.482893] Lustre: 472593:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789058173/real 1789058173] req@0000000028fc727c x1875962977514752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789058189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15681.575596] Lustre: 472593:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789058174/real 1789058174] req@00000000be3b158c x1875962977514880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789058190 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15685.420540] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [15685.606584] Lustre: 472595:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789058178/real 1789058178] req@00000000d0d6ead4 x1875962977515136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789058194 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15690.786520] Lustre: 472595:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789058183/real 1789058183] req@000000006aeaeafe x1875962977515264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789058199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15696.589519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15704.458662] LustreError: 475288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15704.479706] LustreError: 475288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15704.535771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15705.571825] LustreError: 475289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15705.576687] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15706.612636] LustreError: 475288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15709.733902] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [15711.713570] LustreError: 475289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15716.857334] LustreError: 475288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15719.714818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15720.252104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15721.293949] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15726.305765] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [15735.636349] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15742.898821] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15765.985150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15765.988765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15766.004154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15766.015591] Lustre: Skipped 1 previous similar message [15766.496234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15767.531776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15767.542799] Lustre: Skipped 1 previous similar message [15772.647585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15777.761386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15777.779439] Lustre: Skipped 2 previous similar messages [15780.319157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15780.669894] Lustre: server umount lustre-OST0000 complete [15786.978586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15786.994364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15787.020292] Lustre: Skipped 1 previous similar message [15787.026310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15791.349765] Lustre: server umount lustre-MDT0000 complete [15793.120991] LustreError: 499550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15793.135309] LustreError: 499550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15795.752113] LustreError: 474118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058304 with bad export cookie 4350244390571670545 [15795.768921] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15795.771519] LustreError: 474118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15796.247918] Lustre: server umount lustre-MDT0001 complete [15809.884605] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [15813.323738] Key type lgssc unregistered [15813.970486] LNet: 502350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15813.991447] LNetError: 502350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15814.003895] LNet: Removed LNI 192.168.202.159@tcp [15814.011565] LNet: Removed LNI 192.168.202.159@tcp10 [15814.025409] LNet: Removed LNI 192.168.202.159@tcp11 [15814.030255] LNet: Removed LNI 192.168.202.159@tcp12 [15814.046176] LNet: Removed LNI 192.168.202.159@tcp13 [15814.053524] LNet: Removed LNI 192.168.202.159@tcp14 [15814.057614] LNet: Removed LNI 192.168.202.159@tcp15 [15814.065416] LNet: Removed LNI 192.168.202.159@tcp16 [15814.071946] LNet: Removed LNI 192.168.202.159@tcp17 [15814.075015] LNet: Removed LNI 192.168.202.159@tcp18 [15814.083621] LNet: Removed LNI 192.168.202.159@tcp19 [15814.088817] LNet: Removed LNI 192.168.202.159@tcp20 [15814.091892] LNet: Removed LNI 192.168.202.159@tcp21 [15814.099785] LNet: Removed LNI 192.168.202.159@tcp22 [15814.106912] LNet: Removed LNI 192.168.202.159@tcp23 [15814.115486] LNet: Removed LNI 192.168.202.159@tcp24 [15814.121338] LNet: Removed LNI 192.168.202.159@tcp25 [15814.136068] LNet: Removed LNI 192.168.202.159@tcp26 [15814.151346] LNet: Removed LNI 192.168.202.159@tcp27 [15814.154292] LNet: Removed LNI 192.168.202.159@tcp28 [15814.156610] LNet: Removed LNI 192.168.202.159@tcp29 [15814.159857] LNet: Removed LNI 192.168.202.159@tcp30 [15814.163769] LNet: Removed LNI 192.168.202.159@tcp31 [15814.167040] LNet: Removed LNI 192.168.202.159@tcp32 [15814.174357] LNet: Removed LNI 192.168.202.159@tcp33 [15814.175993] LNet: Removed LNI 192.168.202.159@tcp34 [15814.177715] LNet: Removed LNI 192.168.202.159@tcp35 [15814.189100] LNet: Removed LNI 192.168.202.159@tcp36 [15814.196491] LNet: Removed LNI 192.168.202.159@tcp37 [15814.202425] LNet: Removed LNI 192.168.202.159@tcp38 [15814.212428] LNet: Removed LNI 192.168.202.159@tcp39 [15814.216856] LNet: Removed LNI 192.168.202.159@tcp40 [15814.227243] LNet: Removed LNI 192.168.202.159@tcp41 [15814.235784] LNet: Removed LNI 192.168.202.159@tcp42 [15814.245870] LNet: Removed LNI 192.168.202.159@tcp43 [15814.252301] LNet: Removed LNI 192.168.202.159@tcp44 [15814.255841] LNet: Removed LNI 192.168.202.159@tcp45 [15814.266584] LNet: Removed LNI 192.168.202.159@tcp46 [15814.270439] LNet: Removed LNI 192.168.202.159@tcp47 [15814.276094] LNet: Removed LNI 192.168.202.159@tcp48 [15814.280203] LNet: Removed LNI 192.168.202.159@tcp49 [15814.298142] LNet: Removed LNI 192.168.202.159@tcp50 [15815.329207] Key type .llcrypt unregistered [15815.331746] Key type ._llcrypt unregistered [15838.529174] Key type ._llcrypt registered [15838.535365] Key type .llcrypt registered [15838.703494] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [15858.120887] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [15860.470575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15860.500725] alg: No test for adler32 (adler32-zlib) [15861.617130] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [15861.954983] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [15863.703536] Key type lgssc registered [15865.087368] Lustre: Echo OBD driver; http://www.lustre.org/ [15903.714900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15903.759766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15905.341409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15905.433983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15905.558275] Lustre: lustre-MDT0000: new disk, initializing [15905.629146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15905.671877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15911.996625] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [15924.118047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15924.392857] Lustre: 505543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15924.502952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15924.514943] Lustre: Skipped 1 previous similar message [15924.694761] Lustre: lustre-MDT0001: new disk, initializing [15924.878346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15924.948246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15924.970813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15932.605334] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [15944.728879] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15950.678940] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15961.164024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15961.256248] Lustre: *** cfs_fail_loc=90f, val=0*** [15961.290939] Lustre: lustre-OST0000: new disk, initializing [15961.294522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15961.298792] Lustre: 507561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15961.343483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15968.980729] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [15971.415422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15971.425905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15971.582800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15981.869157] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15997.410661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15997.411425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15997.438926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15997.451873] Lustre: Skipped 1 previous similar message [15997.476190] Lustre: Skipped 1 previous similar message [16001.202221] Lustre: server umount lustre-OST0000 complete [16006.624258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16006.635883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16006.658844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16007.650471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16007.671235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16007.673591] Lustre: Skipped 1 previous similar message [16012.774278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16013.046210] Lustre: server umount lustre-MDT0000 complete [16017.675409] LustreError: 505537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058526 with bad export cookie 17488231462305682492 [16017.675873] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16017.680313] LustreError: 505537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16018.000525] Lustre: server umount lustre-MDT0001 complete [16029.919404] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [16033.108330] Key type lgssc unregistered [16033.566288] LNet: 509471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16033.605675] LNetError: 509471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16033.628632] LNet: Removed LNI 192.168.202.159@tcp [16035.124261] Key type .llcrypt unregistered [16035.131174] Key type ._llcrypt unregistered [16061.231371] Key type ._llcrypt registered [16061.234619] Key type .llcrypt registered [16061.332951] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [16062.411443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16062.627726] alg: No test for adler32 (adler32-zlib) [16063.890654] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [16064.435353] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [16066.175197] Key type lgssc registered [16067.782988] Lustre: Echo OBD driver; http://www.lustre.org/ [16078.232909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16078.311283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16080.055267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16087.231846] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [16090.603135] LustreError: 510739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16090.644133] LustreError: 510739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16095.722142] LustreError: 510740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16099.503864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16099.821654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16106.600681] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [16114.956739] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16122.453440] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16131.419152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16131.924482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16139.109767] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [16149.079594] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16166.624522] loop0: detected capacity change from 0 to 204800000 [16166.928537] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16166.953403] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16166.973732] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16174.922835] loop0: detected capacity change from 0 to 204800000 [16175.133558] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16175.147850] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16175.169062] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16183.074605] loop0: detected capacity change from 0 to 204800000 [16190.490423] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16190.651811] Lustre: fs15246-OST0000: new disk, initializing [16190.656490] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16190.661705] Lustre: 514338:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16190.711296] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16198.933653] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [16210.161638] loop1: detected capacity change from 0 to 204800000 [16217.079417] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16217.273620] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16217.340868] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16217.437569] Lustre: fs15246-MDT0000: new disk, initializing [16217.623480] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16217.684989] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16217.875317] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16218.914375] Lustre: 515460:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a3c769bb x1875963882296704/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789058744 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16222.808039] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [16223.980181] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16223.993649] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16224.046385] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16249.347487] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16249.381623] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16252.481375] Lustre: server umount fs15246-MDT0000 complete [16254.433406] LustreError: 513296:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16259.573253] LustreError: 510734:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16262.158932] Lustre: server umount fs15246-OST0000 complete [16277.472590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16277.493334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16277.548886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16278.499676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16278.522689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16278.555409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16285.169498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16285.172426] Lustre: Skipped 2 previous similar messages [16290.276386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16290.284538] Lustre: Skipped 1 previous similar message [16291.295645] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16291.432833] Lustre: server umount lustre-OST0000 complete [16299.497589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16299.504760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16299.538955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16311.263150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16311.603203] Lustre: server umount lustre-MDT0000 complete [16315.880453] LustreError: 510740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16315.895198] LustreError: 510740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16316.146392] LustreError: 510719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058825 with bad export cookie 13297463876177013223 [16316.169440] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16316.174620] LustreError: 510719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16316.619203] Lustre: server umount lustre-MDT0001 complete [16330.626724] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [16333.779790] Key type lgssc unregistered [16334.049091] LNet: 518560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16334.060458] LNetError: 518560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16335.077893] LNet: Removed LNI 192.168.202.159@tcp [16336.363857] Key type .llcrypt unregistered [16336.365899] Key type ._llcrypt unregistered [16358.518108] Key type ._llcrypt registered [16358.521048] Key type .llcrypt registered [16358.748663] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [16360.226804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16360.485621] alg: No test for adler32 (adler32-zlib) [16361.710260] Lustre: Lustre: Build Version: 2.17.58_2_g725efe4 [16362.061186] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [16363.833182] Key type lgssc registered [16365.496479] Lustre: Echo OBD driver; http://www.lustre.org/ [16410.962876] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [16425.336370] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [16468.309822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16468.340367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16469.543437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16469.592595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16469.722783] Lustre: lustre-MDT0000: new disk, initializing [16469.946100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16469.975219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16476.249285] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [16487.374668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16487.483854] Lustre: 523026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16487.536549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16487.541211] Lustre: Skipped 1 previous similar message [16487.625851] Lustre: lustre-MDT0001: new disk, initializing [16487.747432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16487.781192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16487.789714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16492.943427] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [16505.653457] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16514.212093] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16523.554977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16524.009102] Lustre: lustre-OST0000: new disk, initializing [16524.034027] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16524.042900] Lustre: 524997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16524.235347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16531.971198] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [16532.066562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16532.087897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16532.226185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16543.336604] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16547.922306] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16548.243781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16553.948159] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16554.280608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16558.065163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16558.066043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16558.071332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16558.080943] LustreError: Skipped 1 previous similar message [16558.092774] Lustre: Skipped 1 previous similar message [16562.341789] Lustre: server umount lustre-OST0000 complete [16568.289184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16568.320498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16568.328844] Lustre: Skipped 2 previous similar messages [16571.705750] Lustre: server umount lustre-MDT0000 complete [16577.855504] LustreError: 523019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059087 with bad export cookie 14566098036806286441 [16577.862074] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16577.886665] LustreError: 523019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16578.826891] Lustre: server umount lustre-MDT0001 complete [16608.218713] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [16620.879283] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [16659.487988] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16368 sec ========= 12:52:46 (1789059166) [16661.759137] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:52:48 (1789059168) === [16665.679566] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:52:53 (1789059173) === [16713.015733] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [16717.333471] Key type lgssc unregistered [16717.791705] LNet: 531645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16717.828220] LNetError: 531645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16717.854160] LNet: Removed LNI 192.168.202.159@tcp [16718.838292] Key type .llcrypt unregistered [16718.843448] Key type ._llcrypt unregistered