[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 595118059 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002000] LSM: Security Framework initializing [ 0.002000] Yama: becoming mindful. [ 0.002000] SELinux: Initializing. [ 0.002000] *** VALIDATE selinux *** [ 0.002000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.002000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.002000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] *** VALIDATE tmpfs *** [ 0.002000] *** VALIDATE proc *** [ 0.002000] *** VALIDATE cgroup *** [ 0.003008] *** VALIDATE cgroup2 *** [ 0.004273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.005188] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.006009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.007028] Spectre V2 : User space: Vulnerable [ 0.008007] Speculative Store Bypass: Vulnerable [ 0.011357] debug: unmapping init [mem 0xffffffffa2859000-0xffffffffa2860fff] [ 0.013263] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.014775] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.015023] ... version: 2 [ 0.016011] ... bit width: 48 [ 0.017011] ... generic registers: 4 [ 0.018012] ... value mask: 0000ffffffffffff [ 0.019014] ... max period: 00007fffffffffff [ 0.020014] ... fixed-purpose events: 3 [ 0.021010] ... event mask: 000000070000000f [ 0.022278] rcu: Hierarchical SRCU implementation. [ 0.024900] smp: Bringing up secondary CPUs ... [ 0.025566] x86: Booting SMP configuration: [ 0.026024] .... node #0, CPUs: #1 #2 #3 [ 0.037801] smp: Brought up 1 node, 4 CPUs [ 0.039011] smpboot: Max logical packages: 1 [ 0.040116] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101290] node 0 deferred pages initialised in 59ms [ 0.104100] devtmpfs: initialized [ 0.105218] x86/mm: Memory block size: 128MB [ 0.107808] gcov: version magic: 0x41383552 [ 0.109293] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.110087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.111378] pinctrl core: initialized pinctrl subsystem [ 0.112192] [ 0.112944] ************************************************************* [ 0.113014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.114010] ** ** [ 0.115011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.116010] ** ** [ 0.117011] ** This means that this kernel is built to expose internal ** [ 0.118058] ** IOMMU data structures, which may compromise security on ** [ 0.119011] ** your system. ** [ 0.120010] ** ** [ 0.121011] ** If you see this message and you are not debugging the ** [ 0.122006] ** kernel, report this immediately to your vendor! ** [ 0.123009] ** ** [ 0.124013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.125016] ************************************************************* [ 0.126726] NET: Registered protocol family 16 [ 0.127507] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.128056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.129068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.130449] cpuidle: using governor menu [ 0.131000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.134567] PCI: Using configuration type 1 for base access [ 0.137125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.147052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.148025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.150085] cryptd: max_cpu_qlen set to 1000 [ 0.152264] ACPI: Added _OSI(Module Device) [ 0.153010] ACPI: Added _OSI(Processor Device) [ 0.154011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.155014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.159308] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.161470] ACPI: Interpreter enabled [ 0.162059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.163010] ACPI: Using IOAPIC for interrupt routing [ 0.164110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.165386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.175718] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.176038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.177018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.178080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.180493] acpiphp: Slot [2] registered [ 0.181140] acpiphp: Slot [5] registered [ 0.182132] acpiphp: Slot [6] registered [ 0.183135] acpiphp: Slot [7] registered [ 0.184107] acpiphp: Slot [8] registered [ 0.185116] acpiphp: Slot [9] registered [ 0.186187] acpiphp: Slot [10] registered [ 0.187141] acpiphp: Slot [3] registered [ 0.188098] acpiphp: Slot [4] registered [ 0.189101] acpiphp: Slot [11] registered [ 0.190096] acpiphp: Slot [12] registered [ 0.191086] acpiphp: Slot [13] registered [ 0.192108] acpiphp: Slot [14] registered [ 0.193092] acpiphp: Slot [15] registered [ 0.194099] acpiphp: Slot [16] registered [ 0.195098] acpiphp: Slot [17] registered [ 0.196101] acpiphp: Slot [18] registered [ 0.197102] acpiphp: Slot [19] registered [ 0.198075] acpiphp: Slot [20] registered [ 0.199067] acpiphp: Slot [21] registered [ 0.200102] acpiphp: Slot [22] registered [ 0.201165] acpiphp: Slot [23] registered [ 0.202097] acpiphp: Slot [24] registered [ 0.203062] acpiphp: Slot [25] registered [ 0.204114] acpiphp: Slot [26] registered [ 0.205068] acpiphp: Slot [27] registered [ 0.206100] acpiphp: Slot [28] registered [ 0.207119] acpiphp: Slot [29] registered [ 0.208057] acpiphp: Slot [30] registered [ 0.209129] acpiphp: Slot [31] registered [ 0.210081] PCI host bridge to bus 0000:00 [ 0.211015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.214020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.216019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.219025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.223034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.226020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.227121] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.230009] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.233324] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.244843] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.250057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.252023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.255019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.257018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.260349] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.263927] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.267047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.271692] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.278014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.293013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.300028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.310305] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.325015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.335013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.361014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.376163] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.383013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.390014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.420015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.430201] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.441014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.449013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.472015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.488140] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.497015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.503015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.526020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.539707] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.548009] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.590999] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.598015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.606018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.634011] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.645816] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.649491] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.652425] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.655366] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.658206] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.664033] iommu: Default domain type: Passthrough [ 0.667320] SCSI subsystem initialized [ 0.669109] ACPI: bus type USB registered [ 0.671126] usbcore: registered new interface driver usbfs [ 0.673087] usbcore: registered new interface driver hub [ 0.675092] usbcore: registered new device driver usb [ 0.677170] pps_core: LinuxPPS API ver. 1 registered [ 0.679008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.682069] PTP clock support registered [ 0.684123] EDAC MC: Ver: 3.0.0 [ 0.686123] PCI: Using ACPI for IRQ routing [ 0.688665] NetLabel: Initializing [ 0.689012] NetLabel: domain hash size = 128 [ 0.690017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.691137] NetLabel: unlabeled traffic allowed by default [ 0.694272] vgaarb: loaded [ 0.696465] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.699016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.705195] clocksource: Switched to clocksource kvm-clock [ 0.813110] VFS: Disk quotas dquot_6.6.0 [ 0.814593] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.818260] *** VALIDATE ramfs *** [ 0.819977] *** VALIDATE hugetlbfs *** [ 0.821752] pnp: PnP ACPI init [ 0.824821] pnp: PnP ACPI: found 6 devices [ 0.841339] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.845133] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.847935] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.850846] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.854136] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.857511] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.861232] NET: Registered protocol family 2 [ 0.863990] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.870386] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.874595] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.880398] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.884419] TCP: Hash tables configured (established 65536 bind 65536) [ 0.887844] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.891454] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.894690] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.898160] NET: Registered protocol family 1 [ 0.901452] RPC: Registered named UNIX socket transport module. [ 0.904623] RPC: Registered udp transport module. [ 0.907094] RPC: Registered tcp transport module. [ 0.909498] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.912922] NET: Registered protocol family 44 [ 0.914986] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.917473] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.919973] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.922568] PCI: CLS 0 bytes, default 64 [ 0.924390] Unpacking initramfs... [ 2.316353] debug: unmapping init [mem 0xffff8d503cc54000-0xffff8d503ffbffff] [ 2.321288] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.323550] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.327049] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.822449] Initialise system trusted keyrings [ 2.823863] Key type blacklist registered [ 2.825481] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.839891] zbud: loaded [ 2.843173] *** VALIDATE nfs *** [ 2.844044] *** VALIDATE nfs4 *** [ 2.845208] pstore: using deflate compression [ 2.848174] Platform Keyring initialized [ 2.957808] NET: Registered protocol family 38 [ 2.959670] Key type asymmetric registered [ 2.961677] Asymmetric key parser 'x509' registered [ 2.964471] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.971681] io scheduler mq-deadline registered [ 2.973083] io scheduler kyber registered [ 2.974608] io scheduler bfq registered [ 2.976223] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.980109] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.982792] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.985356] ACPI: Power Button [PWRF] [ 2.991968] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.999642] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.014111] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.021608] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.045036] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.075477] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.105578] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.111763] Non-volatile memory driver v1.3 [ 3.113467] Linux agpgart interface v0.103 [ 3.147842] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.151020] vda: detected capacity change from 0 to 74825728 [ 3.169325] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.173194] vdb: detected capacity change from 0 to 1073741824 [ 3.189398] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.192667] vdc: detected capacity change from 0 to 2621440000 [ 3.210645] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.216119] vdd: detected capacity change from 0 to 2621440000 [ 3.233739] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.238155] vde: detected capacity change from 0 to 4294967296 [ 3.257289] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.261352] vdf: detected capacity change from 0 to 4294967296 [ 3.270905] libphy: Fixed MDIO Bus: probed [ 3.277674] usbcore: registered new interface driver usbserial_generic [ 3.281358] usbserial: USB Serial support registered for generic [ 3.285579] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.291664] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.294540] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.297764] mousedev: PS/2 mouse device common for all mice [ 3.307130] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.314345] rtc_cmos 00:05: RTC can wake from S4 [ 3.320714] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.322755] rtc_cmos 00:05: registered as rtc0 [ 3.328233] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.329077] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.335586] intel_pstate: CPU model not supported [ 3.339321] hid: raw HID events driver (C) Jiri Kosina [ 3.341831] usbcore: registered new interface driver usbhid [ 3.343281] usbhid: USB HID core driver [ 3.345467] drop_monitor: Initializing network drop monitor service [ 3.347233] Initializing XFRM netlink socket [ 3.348482] NET: Registered protocol family 10 [ 3.350413] Segment Routing with IPv6 [ 3.351323] NET: Registered protocol family 17 [ 3.355424] mpls_gso: MPLS GSO support [ 3.360382] RAS: Correctable Errors collector initialized. [ 3.362932] AVX version of gcm_enc/dec engaged. [ 3.364713] AES CTR mode by8 optimization enabled [ 3.436638] sched_clock: Marking stable (3436561287, 0)->(4487885588, -1051324301) [ 3.440997] registered taskstats version 1 [ 3.443065] Loading compiled-in X.509 certificates [ 3.445784] zswap: loaded using pool lzo/zbud [ 3.473989] Key type big_key registered [ 3.487482] Key type encrypted registered [ 3.489222] ima: No TPM chip found, activating TPM-bypass! [ 3.491086] ima: Allocated hash algorithm: sha1 [ 3.492515] ima: No architecture policies found [ 3.494504] evm: Initialising EVM extended attributes: [ 3.496322] evm: security.selinux [ 3.497455] evm: security.ima [ 3.498477] evm: security.capability [ 3.499791] evm: HMAC attrs: 0x1 [ 3.502720] rtc_cmos 00:05: setting system clock to 2026-08-22 23:05:52 UTC (1787439952) [ 3.509438] debug: unmapping init [mem 0xffffffffa3803000-0xffffffffa39fffff] [ 3.512429] debug: unmapping init [mem 0xffffffffa2582000-0xffffffffa2858fff] [ 3.521226] Write protecting the kernel read-only data: 28672k [ 3.524980] debug: unmapping init [mem 0xffffffffa0c03000-0xffffffffa0dfffff] [ 3.531088] debug: unmapping init [mem 0xffffffffa1514000-0xffffffffa15fffff] [ 3.576469] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.589630] systemd[1]: Detected virtualization kvm. [ 3.593153] systemd[1]: Detected architecture x86-64. [ 3.596283] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.628147] systemd[1]: No hostname configured. [ 3.631222] systemd[1]: Set hostname to . [ 3.634862] random: systemd: uninitialized urandom read (16 bytes read) [ 3.639546] systemd[1]: Initializing machine ID from random generator. [ 3.688656] random: ln: uninitialized urandom read (6 bytes read) [ 3.793431] random: systemd: uninitialized urandom read (16 bytes read) [ 3.797984] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.805571] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.813562] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.641794] device-mapper: uevent: version 1.0.3 [ 4.644707] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.549508] virtio_net virtio0 ens2: renamed from eth0 [ 5.595795] random: fast init done [ 5.647911] scsi host0: ata_piix [ 5.699714] scsi host1: ata_piix [ 5.701278] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.704547] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.006891] random: crng init done [ 10.012531] random: 7 urandom warning(s) missed due to ratelimiting [ 10.732511] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.174778] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. [ 12.307583] hrtimer: interrupt took 4732394 ns Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.880415] printk: systemd: 25 output lines suppressed due to ratelimiting [ 17.170311] SELinux: Disabled at runtime. [ 17.366391] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.389031] systemd[1]: Detected virtualization kvm. [ 17.395139] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.963677] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.980686] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.031760] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.047532] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.066828] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.124594] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.151755] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK [0[ 20.475719] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 22.100333] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.499169] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.580830] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.343670] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.445490] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 30.061523] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 30.983627] NFS: Registering the id_resolver key type [ 30.989471] Key type id_resolver registered [ 30.995180] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 102.094938] libcfs: loading out-of-tree module taints kernel. [ 102.143603] Key type ._llcrypt registered [ 102.146217] Key type .llcrypt registered [ 102.226521] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 118.807616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 120.714743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.740590] alg: No test for adler32 (adler32-zlib) [ 122.557343] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 123.421163] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 125.183209] Key type lgssc registered [ 127.404553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.108531] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.250678] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 201.137596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.205448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.539523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.563083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.680863] Lustre: lustre-MDT0000: new disk, initializing [ 202.818297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.860669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.267618] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.061906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.160165] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.202654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.209041] Lustre: Skipped 1 previous similar message [ 222.321244] Lustre: lustre-MDT0001: new disk, initializing [ 222.447190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.491964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 222.504584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.616480] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.458870] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 242.890589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 243.227375] Lustre: lustre-OST0000: new disk, initializing [ 243.241769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 243.251405] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 243.373539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.506171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.515862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 246.625803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 249.796451] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.379634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.567467] Lustre: lustre-OST0001: new disk, initializing [ 266.573812] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 266.586823] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 266.694447] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.046069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 274.063316] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 274.169963] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 274.753388] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 286.874777] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 297.521438] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.462939] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 310.047479] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 314.287265] Lustre: DEBUG MARKER: Client: 2.17.57.63 [ 316.769604] Lustre: DEBUG MARKER: MDS: 2.17.57.63 [ 319.621165] Lustre: DEBUG MARKER: OSS: 2.17.57.63 [ 321.255284] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 19:11:08 EDT 2026 [ 344.759285] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 346.468750] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 366.052824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 366.061448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.088471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.565480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.590638] Lustre: Skipped 1 previous similar message [ 366.600069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.612648] Lustre: Skipped 2 previous similar messages [ 369.514411] Lustre: server umount lustre-MDT0000 complete [ 373.058128] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787440322 with bad export cookie 11172031861137816256 [ 373.058227] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.073630] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 376.799653] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.801195] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.803305] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.833878] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 376.864880] Lustre: Skipped 2 previous similar messages [ 378.848441] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.848537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 378.873054] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 378.882906] Lustre: Skipped 2 previous similar messages [ 379.566317] Lustre: server umount lustre-MDT0001 complete [ 383.688714] Lustre: server umount lustre-OST0000 complete [ 387.669825] Lustre: server umount lustre-OST0001 complete [ 395.164976] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 406.446671] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 446.633818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.950440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 446.994465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 447.074785] Lustre: lustre-MDT0000: new disk, initializing [ 447.255945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.321499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 453.515051] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 465.305691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.431978] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 465.516782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 465.527035] Lustre: Skipped 1 previous similar message [ 465.618226] Lustre: lustre-MDT0001: new disk, initializing [ 465.735688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 465.792742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 465.811173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 470.116416] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 479.795494] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 486.058367] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 492.738736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 493.004865] Lustre: lustre-OST0000: new disk, initializing [ 493.008758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 493.016797] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 493.122393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.626486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 494.648289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 494.798826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 499.523500] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 508.592127] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 513.583183] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 513.810449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 518.653668] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 518.910754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 525.797421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.800756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.826133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.832552] Lustre: Skipped 1 previous similar message [ 535.007414] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 535.285580] Lustre: server umount lustre-OST0000 complete [ 541.155609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.161059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 541.172765] Lustre: Skipped 2 previous similar messages [ 541.186399] Lustre: Skipped 3 previous similar messages [ 545.236934] Lustre: server umount lustre-MDT0000 complete [ 546.279522] LustreError: 15955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.295307] LustreError: 15955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 548.442888] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787440497 with bad export cookie 11172031861137817782 [ 548.447750] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.454580] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 548.832288] Lustre: server umount lustre-MDT0001 complete [ 553.509574] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:15:00 (1787440500) [ 556.289604] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 558.224521] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:15:05 (1787440505) [ 566.133252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.780567] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.852980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.506582] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 571.873907] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.557079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.174821] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 597.675955] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 605.121483] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 613.081438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 620.396746] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 629.364615] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 651.747887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.751079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 651.760772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 651.769612] Lustre: Skipped 1 previous similar message [ 654.592549] Lustre: server umount lustre-OST0000 complete [ 673.249043] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 673.422890] Lustre: server umount lustre-MDT0000 complete [ 676.320227] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.351772] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 677.471942] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787440626 with bad export cookie 11172031861137818671 [ 677.474288] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.484167] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 677.874993] Lustre: server umount lustre-MDT0001 complete [ 689.933266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 693.216965] Key type lgssc unregistered [ 693.533774] LNet: 23017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.556637] LNetError: 23017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.609246] LNet: Removed LNI 192.168.203.155@tcp [ 694.832127] Key type .llcrypt unregistered [ 694.836370] Key type ._llcrypt unregistered [ 719.010592] Key type ._llcrypt registered [ 719.014879] Key type .llcrypt registered [ 719.123236] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 720.120846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.216159] alg: No test for adler32 (adler32-zlib) [ 721.280904] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 721.636088] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 723.367516] Key type lgssc registered [ 724.690975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 733.362164] loop: module loaded [ 733.443650] loop0: detected capacity change from 0 to 204800000 [ 733.698661] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.709401] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 734.138182] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.364419] loop0: detected capacity change from 0 to 204800000 [ 740.565416] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.580117] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.592482] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 747.157444] loop0: detected capacity change from 0 to 204800000 [ 754.250029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 754.289590] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 755.681865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 755.741450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 755.831266] Lustre: lustre-MDT0000: new disk, initializing [ 755.963966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.985584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 761.561895] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 771.584432] loop1: detected capacity change from 0 to 204800000 [ 777.889034] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 778.310315] Lustre: lustre-OST07c6: new disk, initializing [ 778.313951] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 778.318529] Lustre: Skipped 1 previous similar message [ 778.323648] Lustre: 26208:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 778.415772] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 785.690909] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 788.064815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 788.077860] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 788.133727] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 792.371589] Lustre: 25195:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 792.378059] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 792.382553] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 792.387570] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 792.394860] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 792.407190] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 806.544606] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 806.569524] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 806.582301] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 806.594110] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 806.605490] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 806.616601] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 811.629363] Lustre: 25195:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 811.636710] Lustre: 25195:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 811.644485] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 811.655590] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.668622] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 811.683945] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.705364] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 811.729196] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.739099] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 811.748675] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.757627] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 811.768087] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.961194] Lustre: 26835:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 814.972139] Lustre: 26835:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 814.979050] Lustre: 26835:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 814.995044] Lustre: 26835:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.004071] Lustre: 26835:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 815.015985] Lustre: 26835:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.032159] Lustre: 26835:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 815.038410] Lustre: 26835:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.045426] Lustre: 26835:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 815.051449] Lustre: 26835:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.060106] Lustre: 26835:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 815.068794] Lustre: 26835:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 819.917588] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 819.939891] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 819.951265] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 819.968201] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 819.980223] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 819.992442] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.008979] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 820.016356] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.024753] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 820.027128] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.033642] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 820.038715] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 828.082659] Lustre: 26835:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 828.098488] Lustre: 26835:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 828.108622] Lustre: 26835:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 828.114083] Lustre: 26835:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.120674] Lustre: 26835:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 828.126515] Lustre: 26835:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.133593] Lustre: 26835:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 828.137926] Lustre: 26835:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.146055] Lustre: 26835:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 828.152274] Lustre: 26835:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.156145] Lustre: 26835:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 828.165780] Lustre: 26835:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 880.098061] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.124288] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.145312] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 881.886314] Lustre: server umount lustre-OST07c6 complete [ 891.409965] Lustre: server umount lustre-MDT0000 complete [ 908.075521] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 910.893243] Key type lgssc unregistered [ 911.182862] LNet: 28768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.191458] LNetError: 28768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.211934] LNet: Removed LNI 192.168.203.155@tcp [ 912.158522] Key type .llcrypt unregistered [ 912.161233] Key type ._llcrypt unregistered [ 931.867245] Key type ._llcrypt registered [ 931.878543] Key type .llcrypt registered [ 931.995487] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 933.344237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.398290] alg: No test for adler32 (adler32-zlib) [ 934.397119] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 934.603908] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 936.287473] Key type lgssc registered [ 937.506272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 945.215229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 945.234120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.714525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.301366] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 956.902465] LustreError: 30038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.948764] LustreError: 30038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 959.776449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.149990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 964.627127] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 971.403954] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 976.484323] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 983.561428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.948253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.288736] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 991.228146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 997.778456] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1011.689899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.700985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1011.709628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1011.717134] Lustre: Skipped 1 previous similar message [ 1011.728169] LustreError: Skipped 1 previous similar message [ 1016.801469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.809048] Lustre: Skipped 2 previous similar messages [ 1017.175185] Lustre: server umount lustre-OST0000 complete [ 1021.410178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1021.420805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.435886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.951560] Lustre: server umount lustre-MDT0000 complete [ 1027.043069] LustreError: 30035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.065140] LustreError: 30035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1030.033915] LustreError: 30019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787440979 with bad export cookie 16455552668240255260 [ 1030.045877] LustreError: 30019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1030.045907] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.377267] Lustre: server umount lustre-MDT0001 complete [ 1040.131421] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1042.413380] Key type lgssc unregistered [ 1042.702821] LNet: 33777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.714148] LNetError: 33777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.744632] LNet: Removed LNI 192.168.203.155@tcp [ 1043.551988] Key type .llcrypt unregistered [ 1043.556232] Key type ._llcrypt unregistered [ 1063.773309] Key type ._llcrypt registered [ 1063.775234] Key type .llcrypt registered [ 1063.908727] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1065.131831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1065.193747] alg: No test for adler32 (adler32-zlib) [ 1066.282254] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1066.432517] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1068.079464] Key type lgssc registered [ 1069.045861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1075.960248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1075.984713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.455265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.607336] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1087.976176] LustreError: 35045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1087.990690] LustreError: 35045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1090.726402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.142235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1094.907210] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1101.640101] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1106.847957] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.077448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.364890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.808996] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1122.575318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1129.647329] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1142.246820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.259867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.295149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.265920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.283575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.730524] Lustre: server umount lustre-OST0000 complete [ 1152.483152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1152.497152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.521854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.532624] Lustre: Skipped 1 previous similar message [ 1154.917985] Lustre: server umount lustre-MDT0000 complete [ 1158.515983] LustreError: 36888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441107 with bad export cookie 14451723099862791296 [ 1158.517994] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.526995] LustreError: 36888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1158.923228] Lustre: server umount lustre-MDT0001 complete [ 1170.468984] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1173.988577] Key type lgssc unregistered [ 1174.389565] LNet: 38580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.395634] LNetError: 38580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1174.409299] LNet: Removed LNI 192.168.203.155@tcp [ 1175.507732] Key type .llcrypt unregistered [ 1175.510244] Key type ._llcrypt unregistered [ 1201.295407] Key type ._llcrypt registered [ 1201.302632] Key type .llcrypt registered [ 1201.431454] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1203.045492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1203.133507] alg: No test for adler32 (adler32-zlib) [ 1204.342134] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1204.653481] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1206.391163] Key type lgssc registered [ 1207.275564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1215.480620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1215.514256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.034289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.514786] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1227.233140] LustreError: 39999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.254511] LustreError: 39999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1230.606981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.019796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.432812] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1243.121464] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1250.419325] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1257.129275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1257.356635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.546615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1265.166681] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1274.521069] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1282.020653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1282.024059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.057753] LustreError: 39998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.124971] Lustre: server umount lustre-MDT0000 complete [ 1284.069095] LustreError: 40727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.073858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.082434] LustreError: 40727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1284.105436] Lustre: Skipped 1 previous similar message [ 1286.745838] LustreError: 39978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441235 with bad export cookie 8361151111053312234 [ 1286.766060] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.712683] LustreError: 39995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.739168] LustreError: 39995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1287.762432] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 1289.195179] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.222475] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1291.252355] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1292.848884] LustreError: 39993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.885235] LustreError: 39993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1293.533357] Lustre: server umount lustre-MDT0001 complete [ 1314.663930] Lustre: server umount lustre-OST0000 complete [ 1333.169490] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1336.685928] Key type lgssc unregistered [ 1337.174821] LNet: 43862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.196699] LNetError: 43862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1337.231140] LNet: Removed LNI 192.168.203.155@tcp [ 1338.308208] Key type .llcrypt unregistered [ 1338.311793] Key type ._llcrypt unregistered [ 1363.448704] Key type ._llcrypt registered [ 1363.451790] Key type .llcrypt registered [ 1363.614834] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1364.740551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.841921] alg: No test for adler32 (adler32-zlib) [ 1365.830557] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1365.986909] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1367.631422] Key type lgssc registered [ 1369.072995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1378.778124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1378.811994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.412132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.761300] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1390.563934] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.591988] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1395.681292] LustreError: 45279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.802729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.311588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.885734] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1408.557685] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1413.920097] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1419.600965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1419.868844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.098460] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1426.038941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1431.322531] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1436.643914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.648058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1436.655264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.663076] Lustre: Skipped 1 previous similar message [ 1438.274846] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1438.284235] Lustre: Skipped 1 previous similar message [ 1441.764830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.712221] Lustre: server umount lustre-OST0000 complete [ 1462.253282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.257252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.269240] Lustre: Skipped 1 previous similar message [ 1462.286798] Lustre: Skipped 2 previous similar messages [ 1465.529730] Lustre: server umount lustre-MDT0000 complete [ 1466.403478] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.421028] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1468.455127] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441417 with bad export cookie 9167831881252332667 [ 1468.457601] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.462406] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1474.771529] Lustre: server umount lustre-MDT0001 complete [ 1482.743861] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1485.118674] Key type lgssc unregistered [ 1485.302449] LNet: 48988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.311742] LNetError: 48988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.382575] LNet: Removed LNI 192.168.203.155@tcp [ 1487.122475] Key type .llcrypt unregistered [ 1487.124703] Key type ._llcrypt unregistered [ 1514.076647] Key type ._llcrypt registered [ 1514.079227] Key type .llcrypt registered [ 1514.183564] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1516.047059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1516.084878] alg: No test for adler32 (adler32-zlib) [ 1517.658859] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1517.843316] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1519.487240] Key type lgssc registered [ 1520.956977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1529.802647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1529.851337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.228892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.405835] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1541.614193] LustreError: 50407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.634685] LustreError: 50407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1542.826339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.102549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1547.038676] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1555.072306] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1561.412582] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1568.977926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1569.385719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.671588] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1576.587808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1582.810494] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1591.195975] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1594.036320] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1606.025377] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2219h32m33s [ 1607.317464] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1609.698106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.704341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1609.706944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.719082] Lustre: Skipped 1 previous similar message [ 1610.306800] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1610.315593] Lustre: Skipped 2 previous similar messages [ 1611.360244] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1611.367463] Lustre: Skipped 3 previous similar messages [ 1615.237287] Lustre: server umount lustre-MDT0000 complete [ 1616.417138] LustreError: 52114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.075131] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1622.561318] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1622.577288] LustreError: 50402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1623.236462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.313213] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.504716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.547543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1627.022664] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1629.695154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1629.962932] Lustre: DEBUG MARKER: Wait for df (36808) ... [ 1631.155290] Lustre: DEBUG MARKER: done [ 1634.787175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.794862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1634.797725] Lustre: Skipped 2 previous similar messages [ 1634.804356] Lustre: Skipped 1 previous similar message [ 1639.909562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1640.605996] Lustre: server umount lustre-OST0000 complete [ 1645.028899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.040679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1645.042160] Lustre: Skipped 1 previous similar message [ 1651.105356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1651.119424] Lustre: Skipped 6 previous similar messages [ 1658.335153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1658.573314] Lustre: server umount lustre-MDT0000 complete [ 1663.522202] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.532431] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1663.569581] LustreError: 50389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441612 with bad export cookie 12167736849125141856 [ 1663.577503] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1663.582630] LustreError: 50389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1663.779210] Lustre: server umount lustre-MDT0001 complete [ 1678.154199] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1681.710192] Key type lgssc unregistered [ 1682.044855] LNet: 55336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.059749] LNetError: 55336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.101757] LNet: Removed LNI 192.168.203.155@tcp [ 1683.135393] Key type .llcrypt unregistered [ 1683.137370] Key type ._llcrypt unregistered [ 1731.629642] Key type ._llcrypt registered [ 1731.631846] Key type .llcrypt registered [ 1731.794925] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1733.718762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1733.752485] alg: No test for adler32 (adler32-zlib) [ 1735.205586] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1735.786028] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1737.660753] Key type lgssc registered [ 1739.827887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1749.559662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1749.584344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.018632] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1751.087426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1751.376128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.528597] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1766.030345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.115756] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1766.164097] Lustre: 57548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1766.516235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1772.021637] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1780.206922] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1785.887738] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1792.713901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1792.898845] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1793.110399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1798.759199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1799.017642] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1805.902996] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1812.362223] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2219h36m00s [ 1813.763835] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1819.890777] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1822.132511] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1822.213798] LustreError: 57556:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1842.764848] Lustre: lustre-MDT0000: Client 5293dc0c-08b1-4878-8522-ef5435b7420d (at 192.168.203.55@tcp) reconnecting [ 1862.311795] LustreError: 57556:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1864.604128] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1867.108970] Lustre: DEBUG MARKER: done [ 1875.438544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1875.449926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.469851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1880.435769] Lustre: server umount lustre-OST0000 complete [ 1886.188501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.194889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1886.212509] Lustre: Skipped 2 previous similar messages [ 1886.222788] Lustre: Skipped 1 previous similar message [ 1887.272673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.280876] Lustre: Skipped 2 previous similar messages [ 1891.183905] Lustre: server umount lustre-MDT0000 complete [ 1894.973760] LustreError: 57541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441843 with bad export cookie 5082410035987574000 [ 1894.976417] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1894.988357] LustreError: 57541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1895.518797] Lustre: server umount lustre-MDT0001 complete [ 1908.090251] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1910.902916] Key type lgssc unregistered [ 1911.222634] LNet: 61881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.233625] LNetError: 61881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.260818] LNet: Removed LNI 192.168.203.155@tcp [ 1912.298659] Key type .llcrypt unregistered [ 1912.300880] Key type ._llcrypt unregistered [ 1957.689681] Key type ._llcrypt registered [ 1957.691651] Key type .llcrypt registered [ 1957.839914] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1959.212715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1959.266329] alg: No test for adler32 (adler32-zlib) [ 1960.419416] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1960.694949] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1962.399173] Key type lgssc registered [ 1963.498201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1966.831812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1966.863738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.164743] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1968.187786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1968.334245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.119830] Lustre: Failing over lustre-MDT0000 [ 1969.414944] Lustre: server umount lustre-MDT0000 complete [ 1976.118993] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:38:43 (1787441923) [ 1982.524109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.033838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.895570] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1995.826142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.994242] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1996.032979] Lustre: 64678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1996.293482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2000.581465] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2008.198604] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2013.829251] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2020.658764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.903531] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2021.150629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.193409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2028.071374] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2037.830283] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2044.738220] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2052.577278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2052.587284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.608792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.116415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.125571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.137819] Lustre: Skipped 1 previous similar message [ 2053.162571] Lustre: Skipped 2 previous similar messages [ 2058.212328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.225291] Lustre: Skipped 2 previous similar messages [ 2058.706689] Lustre: server umount lustre-MDT0000 complete [ 2062.221444] LustreError: 64671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442011 with bad export cookie 16896753628954864968 [ 2062.225515] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2062.240309] LustreError: 64671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2062.569411] Lustre: server umount lustre-MDT0001 complete [ 2065.783486] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2071.106152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2079.711668] Lustre: 63605:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787442012/real 1787442012] req@00000000bb35febf x1874268687213696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787442028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2079.785656] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.893774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.808683] Lustre: 63607:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787442016/real 1787442016] req@00000000261fbf68 x1874268687213824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787442032 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2084.831429] Lustre: 63606:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787442017/real 1787442017] req@00000000f64186a7 x1874268687213952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787442033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2087.008580] LustreError: 63603:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fdd845cf x1874268687214720/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2087.828332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.004914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2088.741240] Lustre: 63604:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787442021/real 1787442021] req@0000000061d46c86 x1874268687214208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787442037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2093.866210] Lustre: 63605:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787442026/real 1787442026] req@00000000693f1f8e x1874268687214336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787442042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2094.647089] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2095.076908] LustreError: 66531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.126527] LustreError: 66531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2100.225685] LustreError: 68454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.277886] LustreError: 68454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2105.318859] LustreError: 68454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.364775] LustreError: 68454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2107.743903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.156113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2109.173539] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2115.199757] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2126.295198] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2132.116103] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2140.141142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.155059] Lustre: Skipped 2 previous similar messages [ 2140.163111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.182182] Lustre: Skipped 1 previous similar message [ 2145.261685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.274621] Lustre: Skipped 2 previous similar messages [ 2146.323652] Lustre: server umount lustre-MDT0000 complete [ 2150.380907] LustreError: 68452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.414556] LustreError: 68452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2151.115161] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442100 with bad export cookie 16896753628954867243 [ 2151.119120] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.139767] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2151.381457] Lustre: server umount lustre-MDT0001 complete [ 2158.196347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2167.525867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.871173] Lustre: 63605:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787442104/real 1787442104] req@00000000e7329ca4 x1874268687245696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787442120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2171.911911] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.421568] LustreError: 66530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.535982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.628793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2180.655024] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2180.664573] Lustre: Skipped 1 previous similar message [ 2180.723373] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2187.245523] LustreError: 71136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.259767] LustreError: 71136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2188.255200] Lustre: 63607:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787442120/real 1787442120] req@000000000848f410 x1874268687246464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787442136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2188.314041] Lustre: 63607:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2188.825272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.860381] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2194.445184] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2201.696247] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2207.546951] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2214.884046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2214.885191] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.914925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2214.927504] Lustre: Skipped 1 previous similar message [ 2214.945318] Lustre: Skipped 2 previous similar messages [ 2220.911357] Lustre: server umount lustre-MDT0000 complete [ 2223.018562] LustreError: 66530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.045402] LustreError: 66530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2224.454919] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442173 with bad export cookie 16896753628954868622 [ 2224.460942] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2224.466610] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2224.790842] Lustre: server umount lustre-MDT0001 complete [ 2230.827841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2234.336250] Lustre: DEBUG MARKER: files compared the same [ 2246.609398] Lustre: server umount lustre-OST0000 complete [ 2262.306981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2265.065425] Key type lgssc unregistered [ 2265.345972] LNet: 74412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.354031] LNetError: 74412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.427883] LNet: Removed LNI 192.168.203.155@tcp [ 2267.180341] Key type .llcrypt unregistered [ 2267.182811] Key type ._llcrypt unregistered [ 2292.099472] Key type ._llcrypt registered [ 2292.102073] Key type .llcrypt registered [ 2292.232983] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2293.290502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2293.332653] alg: No test for adler32 (adler32-zlib) [ 2294.718697] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2295.021431] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2296.655142] Key type lgssc registered [ 2298.045846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2307.853142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2307.876040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2309.356472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.104803] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2319.845406] LustreError: 75831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.868991] LustreError: 75831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2324.232604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.553439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2329.866944] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2338.205885] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2344.668927] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2350.963328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2351.235891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.360431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2357.982426] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2366.448315] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2380.260079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2380.271131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.272504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2380.273974] LustreError: Skipped 1 previous similar message [ 2380.321384] Lustre: Skipped 1 previous similar message [ 2384.045689] Lustre: server umount lustre-OST0000 complete [ 2390.496177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.500888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2390.524756] Lustre: Skipped 1 previous similar message [ 2390.540867] Lustre: Skipped 2 previous similar messages [ 2395.005162] Lustre: server umount lustre-MDT0000 complete [ 2395.617273] LustreError: 76562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.644929] LustreError: 76562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2399.145251] LustreError: 75812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442348 with bad export cookie 3074073040221857171 [ 2399.156182] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.162878] LustreError: 75812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2399.545746] Lustre: server umount lustre-MDT0001 complete [ 2411.820946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2414.328385] Key type lgssc unregistered [ 2414.605769] LNet: 79368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.617231] LNetError: 79368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.638460] LNet: Removed LNI 192.168.203.155@tcp [ 2415.397539] Key type .llcrypt unregistered [ 2415.401193] Key type ._llcrypt unregistered [ 2439.979204] Key type ._llcrypt registered [ 2439.982343] Key type .llcrypt registered [ 2440.060530] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2441.089554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2441.182881] alg: No test for adler32 (adler32-zlib) [ 2442.350602] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2442.638888] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2444.343360] Key type lgssc registered [ 2445.460075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2453.672597] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2453.731454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2469.346183] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2474.464705] LustreError: 80758:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2501.151255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2508.480095] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2517.001389] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2525.341096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2525.487334] Lustre: *** cfs_fail_loc=706, val=0*** [ 2525.907130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.039165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2530.648879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2531.299687] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.328539] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2535.400230] LustreError: 81842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.441770] LustreError: 81842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2540.068535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.633670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2545.678942] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2552.949475] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2559.403616] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2562.018350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2562.028366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.050809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2566.113703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2566.119487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2566.123928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.495411] Lustre: server umount lustre-OST0000 complete [ 2571.236513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2571.239678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.252833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2571.261978] Lustre: Skipped 1 previous similar message [ 2571.272747] Lustre: Skipped 2 previous similar messages [ 2576.353679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.368784] Lustre: Skipped 1 previous similar message [ 2577.390981] Lustre: server umount lustre-MDT0000 complete [ 2581.170092] LustreError: 80764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442530 with bad export cookie 4351004732618191214 [ 2581.173079] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.180237] LustreError: 80764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2581.734896] Lustre: server umount lustre-MDT0001 complete [ 2593.545873] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2596.474616] Key type lgssc unregistered [ 2596.800083] LNet: 84375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2596.807605] LNetError: 84375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2596.821746] LNet: Removed LNI 192.168.203.155@tcp [ 2597.788311] Key type .llcrypt unregistered [ 2597.791849] Key type ._llcrypt unregistered [ 2622.624521] Key type ._llcrypt registered [ 2622.630640] Key type .llcrypt registered [ 2622.757468] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2623.561748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2623.600982] alg: No test for adler32 (adler32-zlib) [ 2624.806514] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2625.061923] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2626.767266] Key type lgssc registered [ 2627.792047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2635.180588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2635.207486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.780436] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2649.313066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.898518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2654.774713] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2655.216111] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.226294] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2659.304463] LustreError: 86478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.316159] LustreError: 86478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2664.423653] LustreError: 86482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.437988] LustreError: 86482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2664.731960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2665.035499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.058240] LustreError: 86483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.095506] LustreError: 86483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2673.011025] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2674.150681] LustreError: 86478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.187392] LustreError: 86478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2684.029026] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2684.053610] Lustre: MGS: Client 4a7ce149-6253-4d86-8617-f68d44ec5505 (at 0@lo) reconnecting [ 2684.108725] LustreError: 86493:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2684.476031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.634493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2690.516760] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2705.889396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2705.895934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.905656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2709.996471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2710.003869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2710.020180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.221724] Lustre: server umount lustre-OST0000 complete [ 2720.224292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2720.226759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.239791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2720.239799] Lustre: Skipped 1 previous similar message [ 2720.286971] Lustre: Skipped 1 previous similar message [ 2723.034410] Lustre: server umount lustre-MDT0000 complete [ 2725.351534] LustreError: 88026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.375347] LustreError: 88026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2727.644126] Lustre: server umount lustre-MDT0001 complete [ 2731.279065] Lustre: server umount MGS complete [ 2742.436868] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:51:29 (1787442689) [ 2773.756931] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2786.547593] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2827.186602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.613789] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2842.220377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.464219] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2842.500332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2842.574306] Lustre: lustre-MDT0001: new disk, initializing [ 2842.660032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2842.683465] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2842.691836] Lustre: 93665:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2842.705672] LustreError: 93665:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2847.119739] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2859.022330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2859.292866] Lustre: lustre-OST0000: new disk, initializing [ 2859.301102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2859.306396] Lustre: 94432:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2859.408163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.750651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2866.786567] Lustre: 94782:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000068c0cd1e x1874269384517376/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787442831 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2877.121878] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2884.907394] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2884.919882] Lustre: MGS: Client c3b2b2d0-ea52-41f5-9855-e1b5d428559d (at 0@lo) reconnecting [ 2884.954168] Lustre: 92931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2884.981994] LustreError: 92931:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2885.007885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2885.100228] Lustre: lustre-MDT0000: new disk, initializing [ 2885.248622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.295073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2885.332135] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2886.369512] Lustre: 95502:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000007808ac91 x1874269384524416/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787442851 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2890.426823] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2891.489657] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2895.467127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2895.475510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2895.567769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2921.441603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2921.456462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.491074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2921.498490] Lustre: Skipped 1 previous similar message [ 2925.536821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2925.546613] Lustre: Skipped 1 previous similar message [ 2926.776536] Lustre: server umount lustre-OST0000 complete [ 2935.785501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2935.788530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.802338] LustreError: Skipped 1 previous similar message [ 2935.804912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2935.804919] Lustre: Skipped 1 previous similar message [ 2935.845401] Lustre: Skipped 2 previous similar messages [ 2937.227826] Lustre: server umount lustre-MDT0000 complete [ 2940.896687] LustreError: 95766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.930219] LustreError: 95766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2947.573442] Lustre: server umount lustre-MDT0001 complete [ 2951.793483] Lustre: server umount MGS complete [ 2967.817905] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:55:15 (1787442915) [ 2989.145953] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2992.193946] Key type lgssc unregistered [ 2992.519531] LNet: 98004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2992.523923] LNetError: 98004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2993.579585] LNet: Removed LNI 192.168.203.155@tcp [ 2994.536121] Key type .llcrypt unregistered [ 2994.538229] Key type ._llcrypt unregistered [ 3012.827061] Key type ._llcrypt registered [ 3012.831994] Key type .llcrypt registered [ 3012.927592] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3014.094392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3014.258520] alg: No test for adler32 (adler32-zlib) [ 3015.310707] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3015.522376] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3017.183179] Key type lgssc registered [ 3018.468857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3026.345398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3026.376050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3027.955459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.331245] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3037.156606] LustreError: 99226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.174140] LustreError: 99226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3040.632874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.195914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3045.402084] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3052.412125] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3058.567670] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3065.477935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.957296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.927638] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3072.109911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3079.559731] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3087.842429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3087.846428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.853497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3087.864466] Lustre: Skipped 1 previous similar message [ 3087.880186] Lustre: Skipped 1 previous similar message [ 3092.960261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3092.966687] Lustre: Skipped 1 previous similar message [ 3098.102071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3098.110578] Lustre: Skipped 1 previous similar message [ 3101.151122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3101.361692] Lustre: server umount lustre-OST0000 complete [ 3107.809021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3107.826581] LustreError: Skipped 1 previous similar message [ 3107.846186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.876211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3111.859183] Lustre: server umount lustre-MDT0000 complete [ 3113.444553] LustreError: 99227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.472059] LustreError: 99227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3115.907225] LustreError: 99206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443064 with bad export cookie 7659426208450349442 [ 3115.923253] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3116.319216] Lustre: server umount lustre-MDT0001 complete [ 3128.430283] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3131.571194] Key type lgssc unregistered [ 3131.972121] LNet: 102765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3131.980502] LNetError: 102765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3132.013553] LNet: Removed LNI 192.168.203.155@tcp [ 3133.092061] Key type .llcrypt unregistered [ 3133.096168] Key type ._llcrypt unregistered [ 3152.153675] Key type ._llcrypt registered [ 3152.155409] Key type .llcrypt registered [ 3152.306832] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3153.015396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3153.069515] alg: No test for adler32 (adler32-zlib) [ 3154.181726] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3154.441575] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3156.155122] Key type lgssc registered [ 3157.443746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3164.196744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3164.203550] LustreError: 103846:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3164.211232] LustreError: 103849:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3164.227600] LustreError: 103846:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3164.236043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.238580] LustreError: 103846:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3164.238604] LustreError: 103846:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3165.844708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.470436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.023784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3179.948918] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3185.788707] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lsmod [ 3188.824028] LustreError: 105057:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3188.838362] LustreError: 105058:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3188.846728] LustreError: 105057:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 3188.857439] LustreError: 105057:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3188.869668] LustreError: 105057:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3188.883126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.289075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.382453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3195.360361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3195.367249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.382747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3196.389204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3196.390052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3196.411708] Lustre: Skipped 1 previous similar message [ 3196.428776] Lustre: Skipped 2 previous similar messages [ 3198.789379] Lustre: server umount lustre-MDT0000 complete [ 3199.525888] LustreError: 103886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.539815] LustreError: 103886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3202.266074] LustreError: 103869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443151 with bad export cookie 6755278581828219317 [ 3202.267373] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3202.276078] LustreError: 103869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3203.679303] LustreError: 104318:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ab105cf3 x1874269939192832/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3203.804598] Lustre: server umount lustre-MDT0001 complete [ 3217.708844] Lustre: server umount lustre-OST0000 complete [ 3227.813922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.443197] LustreError: 106122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.509214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.768909] LustreError: 106123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.144644] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3238.898335] LustreError: 106122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3244.006779] LustreError: 106123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3244.164000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.793983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3249.499861] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3258.862423] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3264.750825] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3272.589863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.978542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.974164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3279.560315] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3287.834832] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3294.688906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.688649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.703245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.710231] LustreError: Skipped 1 previous similar message [ 3294.723633] Lustre: Skipped 1 previous similar message [ 3295.975704] Lustre: server umount lustre-OST0000 complete [ 3299.808861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.816492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.825730] Lustre: Skipped 2 previous similar messages [ 3299.837611] Lustre: Skipped 1 previous similar message [ 3304.930319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.940202] Lustre: Skipped 1 previous similar message [ 3305.792274] Lustre: server umount lustre-MDT0000 complete [ 3308.911219] LustreError: 107967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443257 with bad export cookie 6755278581828220248 [ 3308.913073] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3308.928195] LustreError: 107967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3309.383795] Lustre: server umount lustre-MDT0001 complete [ 3321.024332] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3323.808477] Key type lgssc unregistered [ 3324.117908] LNet: 109674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3324.129308] LNetError: 109674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3324.160252] LNet: Removed LNI 192.168.203.155@tcp [ 3324.885537] Key type .llcrypt unregistered [ 3324.887875] Key type ._llcrypt unregistered [ 3347.126308] Key type ._llcrypt registered [ 3347.128579] Key type .llcrypt registered [ 3347.242618] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3348.474251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3348.482946] alg: No test for adler32 (adler32-zlib) [ 3349.527312] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3349.857675] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3351.575205] Key type lgssc registered [ 3352.645915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3360.124955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3360.157080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.500191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.070612] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3372.004346] LustreError: 111092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.020454] LustreError: 111092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3374.245458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.710352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3379.341845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3386.767260] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3392.473528] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3399.250093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3399.705085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.468131] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3406.843645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3412.618180] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3417.691702] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3422.634885] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3427.297446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3427.299031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.307561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3427.331479] Lustre: Skipped 1 previous similar message [ 3432.417459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3432.425861] Lustre: Skipped 2 previous similar messages [ 3433.616322] Lustre: server umount lustre-OST0000 complete [ 3437.543314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.548369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3437.561409] Lustre: Skipped 1 previous similar message [ 3437.582193] Lustre: Skipped 1 previous similar message [ 3442.661746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.654850] Lustre: server umount lustre-MDT0000 complete [ 3447.518640] LustreError: 112935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443396 with bad export cookie 20638702040976557 [ 3447.525755] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3447.542792] LustreError: 112935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3447.973616] Lustre: server umount lustre-MDT0001 complete [ 3459.611984] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3462.746769] Key type lgssc unregistered [ 3463.044604] LNet: 114732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3463.053473] LNetError: 114732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3463.085106] LNet: Removed LNI 192.168.203.155@tcp [ 3464.108177] Key type .llcrypt unregistered [ 3464.111694] Key type ._llcrypt unregistered [ 3485.934269] Key type ._llcrypt registered [ 3485.941438] Key type .llcrypt registered [ 3486.149915] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3487.628101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3487.659455] alg: No test for adler32 (adler32-zlib) [ 3488.785620] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3489.126697] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3490.951258] Key type lgssc registered [ 3492.030633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3501.488698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3501.539730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.070231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.267724] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3512.291144] LustreError: 115954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.314717] LustreError: 115954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3515.778688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.047310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3519.958684] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3528.127867] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3533.271857] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3539.989308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3540.302534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.635427] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3547.508143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3552.270060] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3561.962922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3561.974632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.989438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3562.980345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3562.993137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3563.014477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.259919] Lustre: server umount lustre-OST0000 complete [ 3572.705502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3572.710342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3572.719961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3572.725564] Lustre: Skipped 1 previous similar message [ 3577.452340] Lustre: server umount lustre-MDT0000 complete [ 3578.337938] LustreError: 117018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.356199] LustreError: 117018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3581.605143] LustreError: 115936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443530 with bad export cookie 7063225429137821081 [ 3581.611722] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3581.613744] LustreError: 115936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3582.234360] Lustre: server umount lustre-MDT0001 complete [ 3593.137107] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3595.747549] Key type lgssc unregistered [ 3596.058859] LNet: 119491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3596.065277] LNetError: 119491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3596.084651] LNet: Removed LNI 192.168.203.155@tcp [ 3596.987542] Key type .llcrypt unregistered [ 3596.989403] Key type ._llcrypt unregistered [ 3620.287419] Key type ._llcrypt registered [ 3620.293582] Key type .llcrypt registered [ 3620.434533] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3621.741313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3622.000896] alg: No test for adler32 (adler32-zlib) [ 3623.284442] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3623.525623] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3625.223244] Key type lgssc registered [ 3626.628125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3635.100074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3635.135058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.592394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.222243] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3646.950276] LustreError: 120910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.970842] LustreError: 120910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3649.501664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.871139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3653.396978] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3659.616969] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3664.794513] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3670.681141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3671.027307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3676.700914] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3681.274286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3684.361321] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3691.995912] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3696.245355] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3696.254324] Lustre: Skipped 2 previous similar messages [ 3699.245888] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3699.290052] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3700.514280] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3700.525615] Lustre: Skipped 2 previous similar messages [ 3705.406038] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3705.410891] Lustre: Skipped 1 previous similar message [ 3711.196251] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3711.211801] Lustre: Skipped 2 previous similar messages [ 3713.193762] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp [ 3713.216490] Lustre: Skipped 1 previous similar message [ 3717.233794] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.55/24@tcp in log lustre-MDT0000 [ 3717.246130] Lustre: Skipped 2 previous similar messages [ 3719.731768] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3725.281877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3725.291831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.311589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3727.336074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3727.337195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3727.363505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.672780] Lustre: server umount lustre-OST0000 complete [ 3737.057856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3737.063158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.075453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.082608] Lustre: Skipped 1 previous similar message [ 3742.696789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3742.707951] Lustre: Skipped 1 previous similar message [ 3747.808966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3747.823080] Lustre: Skipped 1 previous similar message [ 3749.343130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3749.584581] Lustre: server umount lustre-MDT0000 complete [ 3752.936943] LustreError: 120905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.974661] LustreError: 120905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3753.632662] LustreError: 120890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443702 with bad export cookie 17028195098307660791 [ 3753.639184] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3753.649132] LustreError: 120890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3753.952265] Lustre: server umount lustre-MDT0001 complete [ 3764.984585] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3767.804474] Key type lgssc unregistered [ 3768.120726] LNet: 125717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3768.125895] LNetError: 125717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3768.133572] LNet: Removed LNI 192.168.203.155@tcp [ 3769.199420] Key type .llcrypt unregistered [ 3769.202708] Key type ._llcrypt unregistered [ 3797.169766] Key type ._llcrypt registered [ 3797.172798] Key type .llcrypt registered [ 3797.296771] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3798.464436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3798.536138] alg: No test for adler32 (adler32-zlib) [ 3799.645910] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3799.908568] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3801.567987] Key type lgssc registered [ 3802.638432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3810.356643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3810.375441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.850318] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3811.880081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.888290] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.55/24@tcp [ 3816.352837] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3821.028127] LustreError: 127290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3821.068167] LustreError: 127290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3825.605444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3826.015738] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3826.048725] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3826.104275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3831.724372] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3840.236301] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3846.044896] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3852.458955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.869963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.951258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3859.551109] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3867.570557] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3882.466568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3882.474530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3882.487901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3885.548857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3885.554081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.556485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3885.583836] Lustre: Skipped 1 previous similar message [ 3886.885764] Lustre: server umount lustre-OST0000 complete [ 3890.662791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3890.675272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.681079] Lustre: Skipped 1 previous similar message [ 3895.785842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3895.809916] Lustre: Skipped 2 previous similar messages [ 3896.966731] Lustre: server umount lustre-MDT0000 complete [ 3900.896769] LustreError: 127290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3900.927654] LustreError: 127290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3900.966764] LustreError: 127271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443849 with bad export cookie 14470272835779043729 [ 3900.974115] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3900.985110] LustreError: 127271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3901.777558] Lustre: server umount lustre-MDT0001 complete [ 3917.426822] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3920.741748] Key type lgssc unregistered [ 3921.177185] LNet: 130884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3921.192383] LNetError: 130884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3921.222331] LNet: Removed LNI 192.168.203.155@tcp [ 3922.289911] Key type .llcrypt unregistered [ 3922.292502] Key type ._llcrypt unregistered [ 3951.709399] Key type ._llcrypt registered [ 3951.711580] Key type .llcrypt registered [ 3951.806292] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3966.882866] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3968.204863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3968.234102] alg: No test for adler32 (adler32-zlib) [ 3969.317961] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 3969.514764] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3971.161108] Key type lgssc registered [ 3972.436937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4007.909150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4007.942672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4009.214825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4009.271820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4009.377085] Lustre: lustre-MDT0000: new disk, initializing [ 4009.456072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.474198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4013.528479] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4024.516221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4024.622767] Lustre: 134286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4024.673032] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4024.677147] Lustre: Skipped 1 previous similar message [ 4024.759103] Lustre: lustre-MDT0001: new disk, initializing [ 4024.780076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4024.847103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4024.881289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4024.906190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4029.378885] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4040.014805] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4045.493203] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4051.734296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4051.986772] Lustre: lustre-OST0000: new disk, initializing [ 4051.990500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4051.996207] Lustre: 136252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4052.064394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.448309] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4061.255605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4061.268491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4061.411953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4067.574783] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4071.638763] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4071.851382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4076.250578] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4076.466946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4082.149134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.159219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4082.181969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.189421] Lustre: Skipped 1 previous similar message [ 4084.601394] Lustre: server umount lustre-OST0000 complete [ 4091.364905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4091.376841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.387103] Lustre: Skipped 1 previous similar message [ 4091.392344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4092.388943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.506504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.514785] Lustre: Skipped 3 previous similar messages [ 4102.625301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4102.633493] Lustre: Skipped 1 previous similar message [ 4103.135184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4103.365646] Lustre: server umount lustre-MDT0000 complete [ 4106.702644] LustreError: 134276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444055 with bad export cookie 4684151345366436882 [ 4106.714359] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4106.991518] Lustre: server umount lustre-MDT0001 complete [ 4115.135570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.866542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.319813] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4121.060463] LustreError: 138472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.077520] LustreError: 138472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4125.152853] LustreError: 138473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4128.524421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.039821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4134.251233] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4142.514805] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4149.051550] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4156.055549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.556506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4162.788623] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4171.353692] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4176.842761] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4177.109771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4181.466948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4181.719287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4191.507223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4191.686399] Lustre: lustre-OST0001: new disk, initializing [ 4191.690913] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4191.697407] Lustre: 141844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4191.785710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4197.478648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4197.496916] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4197.598554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4197.762218] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4205.933919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4206.138266] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4210.758665] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4211.140705] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4216.628927] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93cbc9a00800.ost_server_uuid 50 [ 4218.149420] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93cbc9a00800.ost_server_uuid in FULL state after 0 sec [ 4228.345708] Lustre: server umount lustre-OST0001 complete [ 4228.588327] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.598261] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4228.611423] LustreError: Skipped 1 previous similar message [ 4228.613674] LustreError: 141043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.625875] LustreError: 141043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4232.672068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4232.678937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4232.698783] Lustre: Skipped 1 previous similar message [ 4232.704075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4233.713975] LustreError: 140295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.730924] LustreError: 140295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4237.774456] Lustre: server umount lustre-OST0000 complete [ 4241.888530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4241.897714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.915352] Lustre: Skipped 1 previous similar message [ 4247.360629] Lustre: server umount lustre-MDT0000 complete [ 4249.056923] LustreError: 138473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.712486] LustreError: 138451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444199 with bad export cookie 4684151345366437764 [ 4250.716221] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4250.721842] LustreError: 138451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4250.949610] Lustre: server umount lustre-MDT0001 complete [ 4266.564393] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4268.940739] Key type lgssc unregistered [ 4269.203254] LNet: 144847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4269.208645] LNetError: 144847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4269.236742] LNet: Removed LNI 192.168.203.155@tcp [ 4270.057695] Key type .llcrypt unregistered [ 4270.060140] Key type ._llcrypt unregistered [ 4303.712213] Key type ._llcrypt registered [ 4303.715352] Key type .llcrypt registered [ 4303.804436] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4315.000686] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4315.811655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4315.898687] alg: No test for adler32 (adler32-zlib) [ 4316.930482] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 4317.129793] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4318.783180] Key type lgssc registered [ 4319.738361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4349.633911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4349.670167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.906756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4350.937526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4351.013859] Lustre: lustre-MDT0000: new disk, initializing [ 4351.084066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.100525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4354.577798] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4364.431818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.540320] Lustre: 148978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4364.591758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4364.595395] Lustre: Skipped 1 previous similar message [ 4364.664438] Lustre: lustre-MDT0001: new disk, initializing [ 4364.746450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4364.787736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4364.796813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4369.113951] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4377.565708] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4382.984199] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4389.542485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4389.766275] Lustre: lustre-OST0000: new disk, initializing [ 4389.770555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4389.776836] Lustre: 150946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4389.843305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4395.737496] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4396.603053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4396.619022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4396.744623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4404.646670] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4408.614378] Lustre: Failing over lustre-OST0000 [ 4408.853780] Lustre: server umount lustre-OST0000 complete [ 4426.969406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4427.232197] Lustre: 147205:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787444359/real 1787444359] req@000000000c2dcebd x1874271158698496/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787444375 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4427.242241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.273271] Lustre: Skipped 1 previous similar message [ 4427.287997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.319691] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4427.870819] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4428.319093] Lustre: 147205:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787444361/real 1787444361] req@000000003b57be33 x1874271158699648/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787444377 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4428.334861] Lustre: 147205:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4432.393128] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4432.394915] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4432.487768] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4433.439085] Lustre: 147202:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787444366/real 1787444366] req@00000000369c1397 x1874271158700800/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787444382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4433.472438] Lustre: 147202:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4435.817214] Lustre: Failing over lustre-MDT0000 [ 4436.061746] Lustre: server umount lustre-MDT0000 complete [ 4436.449326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4436.461560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.484117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.497078] Lustre: Skipped 1 previous similar message [ 4438.495156] Lustre: 147202:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787444371/real 1787444371] req@000000008b6372af x1874271158701824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787444387 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4442.597393] LustreError: 148990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4442.618980] LustreError: 148990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4443.184890] LustreError: 148984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4447.716569] LustreError: 148989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4447.730225] LustreError: 148989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4452.833256] LustreError: 148990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.854801] LustreError: 148990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4453.644920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4453.772886] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4454.042431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.109761] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4454.349333] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4458.102022] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4459.507497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4459.521391] Lustre: Skipped 1 previous similar message [ 4459.544218] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4459.594699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4468.194722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4468.206837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.227171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4469.729925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4469.737344] Lustre: Skipped 1 previous similar message [ 4471.444249] Lustre: server umount lustre-OST0000 complete [ 4474.851064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4474.852510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.870625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4474.890549] Lustre: Skipped 2 previous similar messages [ 4474.902296] Lustre: Skipped 1 previous similar message [ 4479.973737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4479.978718] Lustre: Skipped 1 previous similar message [ 4485.092670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4485.100161] Lustre: Skipped 1 previous similar message [ 4489.183144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4489.307505] Lustre: server umount lustre-MDT0000 complete [ 4490.212714] LustreError: 149972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.234432] LustreError: 149972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4492.472544] LustreError: 148971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444441 with bad export cookie 2243747263186034383 [ 4492.485254] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4492.495296] LustreError: 148971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4492.753273] Lustre: server umount lustre-MDT0001 complete [ 4502.153868] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4504.335316] Key type lgssc unregistered [ 4504.566655] LNet: 154606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4504.571613] LNetError: 154606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4504.580416] LNet: Removed LNI 192.168.203.155@tcp [ 4505.323299] Key type .llcrypt unregistered [ 4505.327420] Key type ._llcrypt unregistered [ 4540.343924] Key type ._llcrypt registered [ 4540.349615] Key type .llcrypt registered [ 4540.442515] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4541.349080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4541.373765] alg: No test for adler32 (adler32-zlib) [ 4542.577634] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 4542.888104] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4544.607286] Key type lgssc registered [ 4545.489344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4552.032854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4552.051902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4553.376187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4556.799981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4563.938595] LustreError: 156126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.954933] LustreError: 156126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4565.557451] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4569.057528] LustreError: 156127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4569.441529] LustreError: 156155:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4569.457248] LustreError: 156155:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4569.466928] LustreError: 156155:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4569.810973] Lustre: server umount lustre-MDT0000 complete [ 4583.458056] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4593.209573] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4632.470754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.787396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4632.818913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4632.969110] Lustre: lustre-MDT0000: new disk, initializing [ 4633.117229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4633.143592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4638.023622] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4648.988672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.118830] Lustre: 159456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4649.178387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4649.183603] Lustre: Skipped 1 previous similar message [ 4649.236690] Lustre: lustre-MDT0001: new disk, initializing [ 4649.309873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4649.341494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4649.349310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4654.042515] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4663.632536] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4669.335766] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4675.335302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4675.598911] Lustre: lustre-OST0000: new disk, initializing [ 4675.606258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4675.614963] Lustre: 161428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4675.716375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.149913] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4684.834367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4684.843744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4684.944803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4689.518173] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4693.783854] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4695.897789] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:24:03 (1787444643) [ 4941.280092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.283531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4941.283578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4941.285547] Lustre: Skipped 1 previous similar message [ 4941.304106] Lustre: Skipped 2 previous similar messages [ 4945.670402] Lustre: server umount lustre-MDT0000 complete [ 4946.403355] LustreError: 159466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4946.443200] LustreError: 159466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4949.452054] LustreError: 159451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444898 with bad export cookie 8939158100154508243 [ 4949.461476] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4949.487263] LustreError: 159451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4951.520042] LustreError: 162146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4951.534308] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4951.571452] Lustre: Skipped 1 previous similar message [ 4951.577098] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4954.528746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4955.815629] Lustre: server umount lustre-MDT0001 complete [ 4965.190765] Lustre: server umount lustre-OST0000 complete [ 4974.335151] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4984.095242] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5016.487086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5016.765223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5016.794411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5016.858713] Lustre: lustre-MDT0000: new disk, initializing [ 5016.917930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5016.943877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5021.131217] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5030.683213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.832284] Lustre: 166086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5030.901984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5030.907622] Lustre: Skipped 1 previous similar message [ 5031.017101] Lustre: lustre-MDT0001: new disk, initializing [ 5031.107763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5031.146264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5031.163692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5035.904121] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5044.837478] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5050.221971] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5055.938827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.165598] Lustre: lustre-OST0000: new disk, initializing [ 5056.169623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5056.175684] Lustre: 168055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5056.270254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.764697] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5064.233934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5064.241776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5064.303441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5070.765923] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5074.604343] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5074.808801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5079.067227] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5079.321699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5084.650441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5084.658431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5084.668349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5084.685311] Lustre: Skipped 1 previous similar message [ 5084.695113] Lustre: Skipped 1 previous similar message [ 5089.761395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.774508] Lustre: Skipped 1 previous similar message [ 5095.391314] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5095.582991] Lustre: server umount lustre-OST0000 complete [ 5100.003110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.004205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5100.004214] Lustre: Skipped 2 previous similar messages [ 5100.040982] Lustre: Skipped 1 previous similar message [ 5113.311163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5113.552892] Lustre: server umount lustre-MDT0000 complete [ 5115.365237] LustreError: 169393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.411472] LustreError: 169393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5117.557979] LustreError: 166080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445066 with bad export cookie 8939158100154667073 [ 5117.561392] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5117.582513] LustreError: 166080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5118.058101] Lustre: server umount lustre-MDT0001 complete [ 5128.403568] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:31:15 (1787445075) [ 5143.735390] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5153.201153] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5189.947111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.247153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5190.284403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5190.338548] Lustre: lustre-MDT0000: new disk, initializing [ 5190.478164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5190.503959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5195.186187] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5206.568291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5206.734966] Lustre: 172814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5206.978418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5211.439032] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5220.909552] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5228.311620] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5235.833041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5236.186170] Lustre: lustre-OST0000: new disk, initializing [ 5236.189926] Lustre: Skipped 1 previous similar message [ 5236.195441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5236.201669] Lustre: Skipped 2 previous similar messages [ 5236.206761] Lustre: 174790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5243.043064] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5246.563722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5246.584777] Lustre: Skipped 1 previous similar message [ 5246.595854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5246.745500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5252.778155] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5262.305835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.312483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5262.343351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5262.358901] Lustre: Skipped 6 previous similar messages [ 5265.173618] Lustre: server umount lustre-OST0000 complete [ 5272.546632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.564207] Lustre: Skipped 1 previous similar message [ 5276.258645] Lustre: server umount lustre-MDT0000 complete [ 5277.666351] LustreError: 174010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5277.690360] LustreError: 174010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5280.314654] LustreError: 172807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445229 with bad export cookie 8939158100154667955 [ 5280.321599] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5280.323435] LustreError: 172807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5280.623336] Lustre: server umount lustre-MDT0001 complete [ 5291.364796] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:33:58 (1787445238) [ 5308.766819] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5319.951771] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5355.601657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.759379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5355.780806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5355.859623] Lustre: lustre-MDT0000: new disk, initializing [ 5355.941402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5355.950142] Lustre: Skipped 2 previous similar messages [ 5355.966710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5360.088751] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5370.128494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5370.288337] Lustre: 179114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5370.600643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5374.622312] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5383.094729] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5387.801762] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5393.778983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5394.044426] Lustre: 181089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5398.971600] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5400.669711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5406.314434] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5415.909224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5415.914834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.920848] LustreError: Skipped 1 previous similar message [ 5415.924264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5415.942779] Lustre: Skipped 2 previous similar messages [ 5415.963694] Lustre: Skipped 3 previous similar messages [ 5418.240949] Lustre: server umount lustre-OST0000 complete [ 5422.049632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5427.715675] Lustre: server umount lustre-MDT0000 complete [ 5431.081443] LustreError: 179109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445380 with bad export cookie 8939158100154669040 [ 5431.082791] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5431.092415] LustreError: 179109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5431.486238] Lustre: server umount lustre-MDT0001 complete [ 5442.094452] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5444.566883] Key type lgssc unregistered [ 5444.802614] LNet: 182952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5444.809104] LNetError: 182952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5444.822579] LNet: Removed LNI 192.168.203.155@tcp [ 5445.552341] Key type .llcrypt unregistered [ 5445.556304] Key type ._llcrypt unregistered [ 5468.002697] Key type ._llcrypt registered [ 5468.005824] Key type .llcrypt registered [ 5468.140893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5469.242574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5469.441199] alg: No test for adler32 (adler32-zlib) [ 5470.553526] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 5470.817117] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5472.600732] Key type lgssc registered [ 5474.371574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5485.818503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5485.850378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.717130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5493.601089] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5497.828523] LustreError: 184369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5497.874052] LustreError: 184369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5502.884849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.945120] LustreError: 184370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.172177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5506.889125] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5514.435957] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5519.790934] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5526.496575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5526.799844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.526263] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5533.003280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5539.529644] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5558.759323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.765723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5558.767923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5558.783126] Lustre: Skipped 1 previous similar message [ 5558.811500] Lustre: Skipped 1 previous similar message [ 5563.390755] Lustre: server umount lustre-OST0000 complete [ 5568.994867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.019028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5569.030659] Lustre: Skipped 1 previous similar message [ 5572.981497] Lustre: server umount lustre-MDT0000 complete [ 5574.116942] LustreError: 185612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.135691] LustreError: 185612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5576.350277] LustreError: 184351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445525 with bad export cookie 10139842404573403756 [ 5576.357661] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5576.367237] LustreError: 184351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5576.754435] Lustre: server umount lustre-MDT0001 complete [ 5587.536653] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5589.823572] Key type lgssc unregistered [ 5590.144537] LNet: 187926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5590.157814] LNetError: 187926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5590.184519] LNet: Removed LNI 192.168.203.155@tcp [ 5591.033956] Key type .llcrypt unregistered [ 5591.039989] Key type ._llcrypt unregistered [ 5614.451557] Key type ._llcrypt registered [ 5614.453750] Key type .llcrypt registered [ 5614.555111] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5615.530603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5615.650859] alg: No test for adler32 (adler32-zlib) [ 5616.883426] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 5617.184554] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5618.895186] Key type lgssc registered [ 5620.317120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5629.458127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5629.483371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.880236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.543282] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5640.163561] LustreError: 189347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.185892] LustreError: 189347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5643.731086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.076249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5648.673241] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5656.051154] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5661.583845] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5668.874963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5669.173441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.338224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5675.481518] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5682.868601] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5693.930660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.931614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5693.937462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5693.951118] Lustre: Skipped 1 previous similar message [ 5693.976099] Lustre: Skipped 1 previous similar message [ 5695.321607] Lustre: server umount lustre-OST0000 complete [ 5703.009466] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff93cbc6559000.ost_server_uuid 50 [ 5715.009258] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5729.764178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5729.775086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5729.787667] Lustre: Skipped 1 previous similar message [ 5729.803678] Lustre: Skipped 1 previous similar message [ 5734.885783] LustreError: 190074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.905754] LustreError: 190074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5734.948279] Lustre: server umount lustre-MDT0000 complete [ 5735.907586] LustreError: 189347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5735.932899] LustreError: 189347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5738.524027] LustreError: 189326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445687 with bad export cookie 10629691021396377296 [ 5738.524452] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5738.535373] LustreError: 189326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5738.820223] Lustre: server umount lustre-MDT0001 complete [ 5746.611661] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:41:34 (1787445694) [ 5753.064937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5753.519745] LustreError: 193283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5753.603883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5757.961243] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5758.947788] LustreError: 193284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.320318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5765.647055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5769.245420] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5776.846848] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5781.886960] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5788.754331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5789.133692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5795.910476] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5798.404960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5803.620980] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5810.622871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5810.845450] Lustre: lustre-OST0001: new disk, initializing [ 5810.848903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5810.855145] Lustre: 196126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5810.959756] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5817.045859] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5821.032979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5821.041045] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5821.157444] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5825.317440] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5829.102524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5829.112460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.129382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5831.142185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5831.148645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5831.186791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5833.635690] Lustre: server umount lustre-OST0000 complete [ 5836.330409] LustreError: 195104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.354728] LustreError: 195104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5839.242919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5839.444969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5842.734789] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5842.880268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5857.260671] LustreError: 195104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.263255] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5857.266733] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.267432] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5857.267437] Lustre: Skipped 2 previous similar messages [ 5857.278484] LustreError: 195104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5862.033532] Lustre: server umount lustre-OST0001 complete [ 5867.492949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5867.504845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5867.507530] Lustre: Skipped 2 previous similar messages [ 5867.524539] Lustre: Skipped 2 previous similar messages [ 5871.573983] Lustre: server umount lustre-MDT0000 complete [ 5875.230402] LustreError: 193265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445824 with bad export cookie 10629691021396378367 [ 5875.234982] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5875.249049] LustreError: 193265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5875.534454] Lustre: server umount lustre-MDT0001 complete [ 5899.285964] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:44:06 (1787445846) [ 5905.835138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.084219] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5906.119494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5906.271735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.001411] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5918.525324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5918.634252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5918.673629] Lustre: 199558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5923.500616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5931.008308] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5936.751455] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5942.951436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5943.215176] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5949.004584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5949.079265] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5955.838451] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5961.925950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5962.026987] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5968.358298] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5976.474377] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5979.625768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5979.629152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5979.651271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5979.654319] LustreError: Skipped 2 previous similar messages [ 5979.677531] Lustre: Skipped 1 previous similar message [ 5985.874991] Lustre: server umount lustre-OST0000 complete [ 5987.371191] LustreError: 201779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.394472] LustreError: 201779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6005.240088] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6007.865701] Lustre: server umount lustre-OST0001 complete [ 6015.476299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.481385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6015.493228] Lustre: Skipped 4 previous similar messages [ 6015.507541] Lustre: Skipped 6 previous similar messages [ 6026.207134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6026.387181] Lustre: server umount lustre-MDT0000 complete [ 6031.107129] LustreError: 202443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445980 with bad export cookie 10629691021396379774 [ 6031.110147] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6031.129881] LustreError: 202443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6031.536492] Lustre: server umount lustre-MDT0001 complete [ 6060.066734] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:46:47 (1787446007) [ 6076.964922] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 6087.340843] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6124.486534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.862301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6124.913236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6124.992648] Lustre: lustre-MDT0000: new disk, initializing [ 6125.085973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.094271] Lustre: Skipped 3 previous similar messages [ 6125.103760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6130.126814] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6141.715992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.940762] Lustre: 207661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6142.051867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6142.065053] Lustre: Skipped 1 previous similar message [ 6142.198102] Lustre: lustre-MDT0001: new disk, initializing [ 6142.396338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6142.429213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6148.227191] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6158.768501] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6165.183713] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6173.654819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.959394] Lustre: lustre-OST0000: new disk, initializing [ 6173.962706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6173.968679] Lustre: 209631:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6179.927931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6179.937295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6180.040522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6180.388362] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6190.950716] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6195.838934] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6196.048077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6201.470069] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6201.692228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6205.922817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6205.924139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6205.940218] LustreError: Skipped 2 previous similar messages [ 6205.958143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6205.976127] Lustre: Skipped 5 previous similar messages [ 6218.207133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6218.467288] Lustre: server umount lustre-OST0000 complete [ 6236.128144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6236.368934] Lustre: server umount lustre-MDT0000 complete [ 6236.643325] LustreError: 207669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6236.661209] LustreError: 207669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6239.886481] LustreError: 207654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787446188 with bad export cookie 10629691021396381335 [ 6239.895639] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6239.900658] LustreError: 207654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6240.358682] Lustre: server umount lustre-MDT0001 complete [ 6248.580553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.231302] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6264.240470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.873993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6264.882906] Lustre: Skipped 3 previous similar messages [ 6270.445018] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6279.815488] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6286.761882] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6293.817107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6299.383552] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6306.967035] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6313.013531] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6313.317993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6320.025130] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6320.499274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6324.704310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6324.722968] LustreError: Skipped 1 previous similar message [ 6334.947275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6334.950045] Lustre: Skipped 13 previous similar messages [ 6336.991266] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6337.174349] Lustre: server umount lustre-OST0000 complete [ 6343.588340] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6347.879790] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6376.229279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6381.671240] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6387.795783] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6390.952689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.981902] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6390.982787] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.298691] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6401.505088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6401.508366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.525870] LustreError: Skipped 1 previous similar message [ 6401.545098] Lustre: Skipped 5 previous similar messages [ 6404.979222] Lustre: server umount lustre-OST0000 complete [ 6414.967935] Lustre: server umount lustre-MDT0000 complete [ 6416.874175] LustreError: 211854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.905894] LustreError: 211854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6418.861906] LustreError: 211835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787446367 with bad export cookie 10629691021396382217 [ 6418.863630] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6418.870148] LustreError: 211835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6419.323262] Lustre: server umount lustre-MDT0001 complete [ 6427.660398] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:52:55 (1787446375) [ 6433.617141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6438.259592] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6445.671910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.798230] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6457.242428] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6463.019560] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6469.853805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6475.826536] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6481.807849] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6486.016525] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6486.257771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6490.919511] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6491.105263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6498.090349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6498.328509] Lustre: lustre-OST0001: new disk, initializing [ 6498.335513] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6498.351790] Lustre: 221197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6505.551754] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6506.074793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6506.083741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6506.188565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6515.354688] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6520.891175] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6521.160684] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6525.639832] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6525.932551] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6532.064629] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6532.081946] LustreError: Skipped 1 previous similar message [ 6534.520783] Lustre: server umount lustre-OST0001 complete [ 6540.651932] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6540.894226] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6545.072602] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6545.255973] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6574.189727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6574.433694] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6574.442391] Lustre: Skipped 6 previous similar messages [ 6579.685928] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6579.702898] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6579.711209] Lustre: Skipped 1 previous similar message [ 6579.712489] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6580.718145] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6588.681995] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6590.663953] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6595.040913] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6595.049151] Lustre: Skipped 7 previous similar messages [ 6607.327874] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6607.428877] Lustre: server umount lustre-OST0001 complete [ 6630.076982] Lustre: server umount lustre-OST0000 complete [ 6648.287119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6648.727504] Lustre: server umount lustre-MDT0000 complete [ 6652.953072] LustreError: 221196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787446601 with bad export cookie 10629691021396383183 [ 6652.969119] LustreError: 221196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6652.972310] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6659.448642] Lustre: server umount lustre-MDT0001 complete [ 6679.721657] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:57:07 (1787446627) [ 6685.744705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6686.038327] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6686.080179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6689.999631] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6698.274761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6698.367529] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6698.420691] Lustre: 226443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6702.661818] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6708.222363] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6713.443907] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6720.423772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6720.645872] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6726.253806] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6734.564936] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6744.750766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6744.882446] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6753.628426] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6763.188573] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6764.844640] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6769.624610] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6769.923846] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6774.841758] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6775.091848] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6781.231166] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93cbc655f800.ost_server_uuid 50 [ 6782.787094] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93cbc655f800.ost_server_uuid in IDLE state after 0 sec [ 6783.815332] Lustre: Permanently deactivating lustre-OST0001 [ 6783.838484] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6787.372282] Lustre: Permanently reactivating lustre-OST0001 [ 6791.649566] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6791.650707] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.662657] LustreError: Skipped 5 previous similar messages [ 6791.684427] Lustre: Skipped 12 previous similar messages [ 6797.760196] Lustre: server umount lustre-OST0001 complete [ 6801.897716] LustreError: 228288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6801.928302] LustreError: 228288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 6807.504595] Lustre: server umount lustre-OST0000 complete [ 6817.438548] Lustre: server umount lustre-MDT0000 complete [ 6821.177621] LustreError: 226435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787446770 with bad export cookie 10629691021396384471 [ 6821.184496] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6821.194033] LustreError: 226435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6821.710917] Lustre: server umount lustre-MDT0001 complete [ 6846.216869] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:59:53 (1787446793) [ 6852.370728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6852.571469] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6852.599220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6852.611543] Lustre: Skipped 5 previous similar messages [ 6857.392616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6867.040643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.141222] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6867.166193] Lustre: 233134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6872.502924] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6881.441696] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6887.353089] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6894.375519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6894.720324] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6894.762509] Lustre: Permanently deactivating lustre-OST0000 [ 6894.805872] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6902.172989] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6904.338171] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6909.358419] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6916.197868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6916.247759] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6922.307462] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6922.432907] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6929.806917] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6931.750277] Lustre: Permanently reactivating lustre-OST0000 [ 6931.767318] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6931.774130] Lustre: Skipped 2 previous similar messages [ 6931.849797] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6931.863821] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6931.879456] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6931.888556] Lustre: Skipped 1 previous similar message [ 6931.924531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6931.949255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6946.088722] Lustre: server umount lustre-OST0001 complete [ 6955.618738] Lustre: server umount lustre-OST0000 complete [ 6964.819620] Lustre: server umount lustre-MDT0000 complete [ 6967.968748] LustreError: 233126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787446916 with bad export cookie 10629691021396386151 [ 6967.978800] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6967.981670] LustreError: 233126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6968.275519] Lustre: server umount lustre-MDT0001 complete [ 6977.414828] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6979.752724] Key type lgssc unregistered [ 6980.002861] LNet: 238031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6980.033945] LNetError: 238031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6980.059973] LNet: Removed LNI 192.168.203.155@tcp [ 6980.818739] Key type .llcrypt unregistered [ 6980.823172] Key type ._llcrypt unregistered [ 7001.597430] Key type ._llcrypt registered [ 7001.599141] Key type .llcrypt registered [ 7001.712743] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7002.804597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7002.835441] alg: No test for adler32 (adler32-zlib) [ 7003.810363] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7004.006867] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7005.639184] Key type lgssc registered [ 7006.678359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7016.787886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7016.809331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7018.353329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7023.129329] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7028.719835] LustreError: 239497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.748597] LustreError: 239497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7031.073497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.185193] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7031.261110] Lustre: 239486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7031.283860] Lustre: Permanently deactivating lustre-MDT0001 [ 7031.310263] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7031.723555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7035.649724] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7036.953692] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7042.783819] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7047.591907] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7053.093735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.422001] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7057.522780] LustreError: 241338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.544081] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7058.466635] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7063.016777] LustreError: 241338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7064.048935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7065.118552] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7069.156646] LustreError: 241338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7069.181559] LustreError: 241338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7070.881726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7071.171966] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7076.083914] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7076.335144] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7076.350212] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7082.593278] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7086.490750] Lustre: Permanently reactivating lustre-MDT0001 [ 7086.509214] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7086.521652] Lustre: Skipped 2 previous similar messages [ 7091.795307] Lustre: lustre-MDT0001: Client f6342f62-ce43-4e9d-8a2a-07545c961360 (at 192.168.203.55@tcp) reconnecting [ 7096.442283] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.465886] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7096.473235] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7096.483109] LustreError: 243221:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7096.483367] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7096.497132] LustreError: 243221:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7096.497252] LustreError: 243221:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -108 [ 7096.529947] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7096.529947] [ 7096.535631] LustreError: 239489:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7096.810948] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7098.206679] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7102.736243] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7102.909944] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7106.601450] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7106.771737] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7109.099615] Lustre: Permanently deactivating lustre-MDT0001 [ 7109.114508] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7109.119289] Lustre: Skipped 2 previous similar messages [ 7115.352502] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7122.413861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7122.420056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7122.435108] Lustre: Skipped 3 previous similar messages [ 7122.458221] Lustre: Skipped 3 previous similar messages [ 7127.529279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7127.546240] Lustre: Skipped 3 previous similar messages [ 7128.008716] Lustre: server umount lustre-MDT0000 complete [ 7131.185328] LustreError: 243982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787447080 with bad export cookie 7900910083663823685 [ 7131.191052] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7131.200258] LustreError: 243982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7131.593800] Lustre: server umount lustre-MDT0001 complete [ 7140.730422] Lustre: server umount lustre-OST0000 complete [ 7150.163161] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:04:58 (1787447098) [ 7158.420516] Lustre: server umount lustre-OST0001 complete [ 7163.646674] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7172.745342] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7206.547560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.723619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7206.729671] Lustre: Skipped 2 previous similar messages [ 7206.741951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7206.792216] Lustre: lustre-MDT0000: new disk, initializing [ 7206.863036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7206.878698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7211.317957] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7221.686277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.784587] Lustre: 247755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7221.836908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7221.843109] Lustre: Skipped 1 previous similar message [ 7221.969986] Lustre: lustre-MDT0001: new disk, initializing [ 7222.106666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7222.173471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7222.186557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7226.813229] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7237.207172] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7245.028622] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7252.228995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7252.432493] Lustre: lustre-OST0000: new disk, initializing [ 7252.439643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7252.445053] Lustre: 249722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7252.541641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7258.528234] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7262.807733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7262.829373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7262.962235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7268.404417] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7270.447747] LustreError: 247763:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7278.787069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7278.959274] Lustre: lustre-OST0001: new disk, initializing [ 7278.966539] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7278.979159] Lustre: 250837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7280.519415] LustreError: 247763:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7280.544251] LustreError: 250522:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7284.911722] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7285.290885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7285.298417] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7285.395477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7290.584064] LustreError: 250522:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7293.015344] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7294.549582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7295.970445] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7295.979831] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.983187] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7295.997031] Lustre: Skipped 1 previous similar message [ 7299.657064] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 7299.661308] Lustre: Skipped 1 previous similar message [ 7304.750913] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 7304.758264] Lustre: Skipped 2 previous similar messages [ 7310.304377] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7310.422986] Lustre: server umount lustre-OST0001 complete [ 7311.331564] LustreError: 249707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.346278] LustreError: 249707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7321.581386] LustreError: 250249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7321.602943] LustreError: 250249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7326.691446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.709203] Lustre: Skipped 1 previous similar message [ 7326.709224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7326.733274] Lustre: Skipped 4 previous similar messages [ 7332.981326] Lustre: server umount lustre-OST0000 complete [ 7336.931489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.940025] Lustre: Skipped 1 previous similar message [ 7342.871088] Lustre: server umount lustre-MDT0000 complete [ 7346.654026] LustreError: 247746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787447295 with bad export cookie 7900910083663826492 [ 7346.670056] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.677398] LustreError: 247746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7353.109873] Lustre: server umount lustre-MDT0001 complete [ 7365.887484] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7369.707755] Key type lgssc unregistered [ 7370.252174] LNet: 252879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7370.269666] LNetError: 252879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7370.289815] LNet: Removed LNI 192.168.203.155@tcp [ 7371.231960] Key type .llcrypt unregistered [ 7371.234663] Key type ._llcrypt unregistered [ 7410.090210] Key type ._llcrypt registered [ 7410.093272] Key type .llcrypt registered [ 7410.192493] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7411.402931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7411.461528] alg: No test for adler32 (adler32-zlib) [ 7412.626239] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7412.988677] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7414.808042] Key type lgssc registered [ 7416.436533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7425.654897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7425.698474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7427.164400] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7427.195428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7427.487990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7433.015759] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7442.563470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7442.685751] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7442.749579] Lustre: 255066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7442.856856] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7443.048604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7447.151410] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7453.880696] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7458.661000] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7464.087630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7464.299076] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7464.486978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7469.701543] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7472.763344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7476.435996] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7529.440959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7529.453488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.468859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7534.574032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7534.585683] Lustre: Skipped 2 previous similar messages [ 7535.847053] Lustre: server umount lustre-OST0000 complete [ 7539.989994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7552.453827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7553.006875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7559.146630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7559.155073] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7559.155891] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7560.223803] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7567.980507] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7610.346290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7610.352181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.360487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7610.365485] Lustre: Skipped 2 previous similar messages [ 7610.381342] Lustre: Skipped 1 previous similar message [ 7613.818479] Lustre: server umount lustre-OST0000 complete [ 7620.576748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.597996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7623.811990] Lustre: server umount lustre-MDT0000 complete [ 7627.314341] LustreError: 255061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787447576 with bad export cookie 10279005510634758402 [ 7627.319547] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7627.325909] LustreError: 255061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7627.809830] Lustre: server umount lustre-MDT0001 complete [ 7638.791871] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7641.290516] Key type lgssc unregistered [ 7641.557607] LNet: 260492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7641.571104] LNetError: 260492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7641.591680] LNet: Removed LNI 192.168.203.155@tcp [ 7642.392805] Key type .llcrypt unregistered [ 7642.395403] Key type ._llcrypt unregistered [ 7667.116671] Key type ._llcrypt registered [ 7667.119790] Key type .llcrypt registered [ 7667.213269] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7668.183860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7668.207274] alg: No test for adler32 (adler32-zlib) [ 7669.303117] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7669.555094] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7671.264599] Key type lgssc registered [ 7672.493215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7679.637677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7679.700820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7681.147232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7686.004897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7686.629756] LustreError: 261909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.652930] LustreError: 261909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7691.752241] LustreError: 261910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7694.802561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.141813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7700.126612] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7708.222106] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7713.954361] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7720.880879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.166414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7726.336385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7727.438170] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7736.353982] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7766.500314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7766.504787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7766.505921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7766.523892] Lustre: Skipped 1 previous similar message [ 7771.352590] Lustre: server umount lustre-OST0000 complete [ 7776.748474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.757930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7776.777078] Lustre: Skipped 1 previous similar message [ 7776.783409] Lustre: Skipped 1 previous similar message [ 7781.871829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.879869] Lustre: Skipped 2 previous similar messages [ 7786.980962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.987356] Lustre: Skipped 1 previous similar message [ 7790.047235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7790.347724] Lustre: server umount lustre-MDT0000 complete [ 7792.115161] LustreError: 261909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.139756] LustreError: 261909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7794.256491] LustreError: 265462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787447743 with bad export cookie 12173090538327047620 [ 7794.258815] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7794.270075] LustreError: 265462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7794.609242] Lustre: server umount lustre-MDT0001 complete [ 7804.917469] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7807.953367] Key type lgssc unregistered [ 7808.300579] LNet: 266220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7808.310401] LNetError: 266220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7808.333770] LNet: Removed LNI 192.168.203.155@tcp [ 7809.208289] Key type .llcrypt unregistered [ 7809.212099] Key type ._llcrypt unregistered [ 7829.259336] Key type ._llcrypt registered [ 7829.267343] Key type .llcrypt registered [ 7829.388078] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7830.451853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7830.591544] alg: No test for adler32 (adler32-zlib) [ 7831.756680] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7832.054619] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7833.767159] Key type lgssc registered [ 7835.113733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7844.146207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7844.186742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7845.748479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7850.939946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7855.076151] LustreError: 267443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7855.091096] LustreError: 267443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7859.782464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.183959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7864.695363] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7872.736292] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7878.516499] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7886.024167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.461542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7890.568996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7893.513656] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7900.659301] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7916.513487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.514318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7916.538419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7916.552572] Lustre: Skipped 1 previous similar message [ 7921.404078] Lustre: server umount lustre-OST0000 complete [ 7926.752369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7926.753266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.754486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7926.790902] Lustre: Skipped 2 previous similar messages [ 7931.163878] Lustre: server umount lustre-MDT0000 complete [ 7931.880790] LustreError: 267443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7931.893797] LustreError: 267443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7935.405554] LustreError: 267423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787447884 with bad export cookie 3859304238756170923 [ 7935.421452] LustreError: 267423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7935.422436] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7935.807864] Lustre: server umount lustre-MDT0001 complete [ 7945.793292] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7948.241086] Key type lgssc unregistered [ 7948.525791] LNet: 271123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7948.532573] LNetError: 271123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7948.546600] LNet: Removed LNI 192.168.203.155@tcp [ 7949.253396] Key type .llcrypt unregistered [ 7949.255180] Key type ._llcrypt unregistered [ 7972.935351] Key type ._llcrypt registered [ 7972.937591] Key type .llcrypt registered [ 7973.029132] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7974.098232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7974.132334] alg: No test for adler32 (adler32-zlib) [ 7975.213410] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7975.478828] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7977.183194] Key type lgssc registered [ 7978.275893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7986.423905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7986.445732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.010358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.652645] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7997.414894] LustreError: 272542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.426444] LustreError: 272542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8001.215028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.712965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8006.606621] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8014.140322] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8018.995748] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8025.913042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.295450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.447114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8032.567527] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8040.853943] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8068.577537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8068.582602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8068.594200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8068.603533] Lustre: Skipped 1 previous similar message [ 8071.277748] Lustre: server umount lustre-OST0000 complete [ 8078.318219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8078.325692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8078.346620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8078.349757] Lustre: Skipped 1 previous similar message [ 8081.464317] Lustre: server umount lustre-MDT0000 complete [ 8083.943965] LustreError: 272538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.972155] LustreError: 272538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8085.204238] LustreError: 272524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787448034 with bad export cookie 17943801734277137796 [ 8085.220366] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8085.222092] LustreError: 272524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8085.684039] Lustre: server umount lustre-MDT0001 complete [ 8098.061584] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8100.494852] Key type lgssc unregistered [ 8100.809532] LNet: 276853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8100.820187] LNetError: 276853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8101.862241] LNet: Removed LNI 192.168.203.155@tcp [ 8102.756879] Key type .llcrypt unregistered [ 8102.759235] Key type ._llcrypt unregistered [ 8118.511820] Key type ._llcrypt registered [ 8118.514366] Key type .llcrypt registered [ 8118.592564] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8119.361943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8119.374490] alg: No test for adler32 (adler32-zlib) [ 8120.501723] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 8120.779236] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8122.487725] Key type lgssc registered [ 8123.523805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8131.662459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8131.691196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.263748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8138.046074] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8143.847331] LustreError: 278090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.873990] LustreError: 278090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8146.922712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8147.315923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8152.316838] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8159.915812] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8165.996430] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8172.572762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8172.855903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8178.601740] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8178.999630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8186.937196] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8199.664191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8199.670495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8199.692244] LustreError: Skipped 1 previous similar message [ 8199.694772] Lustre: Skipped 1 previous similar message [ 8199.696581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8204.769191] LustreError: 279936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8204.791686] LustreError: 279936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8204.959303] Lustre: server umount lustre-OST0000 complete [ 8209.888650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.891151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8209.918624] Lustre: Skipped 1 previous similar message [ 8209.934948] Lustre: Skipped 2 previous similar messages [ 8215.010247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8215.024771] Lustre: Skipped 1 previous similar message [ 8215.353421] Lustre: server umount lustre-MDT0000 complete [ 8219.105561] LustreError: 278055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787448168 with bad export cookie 8404012655059593469 [ 8219.122856] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8219.129300] LustreError: 278055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8219.566593] Lustre: server umount lustre-MDT0001 complete [ 8232.717806] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8235.882645] Key type lgssc unregistered [ 8236.240794] LNet: 281794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8236.252587] LNetError: 281794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8236.275329] LNet: Removed LNI 192.168.203.155@tcp [ 8237.270474] Key type .llcrypt unregistered [ 8237.278881] Key type ._llcrypt unregistered [ 8249.449466] Key type ._llcrypt registered [ 8249.461067] Key type .llcrypt registered [ 8249.581206] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8270.504269] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8282.289818] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8283.449740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8283.486325] alg: No test for adler32 (adler32-zlib) [ 8284.563635] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 8284.939034] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8286.783201] Key type lgssc registered [ 8288.024925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8319.733754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8319.801782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8321.060426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8321.101146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8321.211962] Lustre: lustre-MDT0000: new disk, initializing [ 8321.347859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8321.371244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8327.022110] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8341.919614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.067964] Lustre: 285328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8342.169531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8342.173817] Lustre: Skipped 1 previous similar message [ 8342.234535] Lustre: lustre-MDT0001: new disk, initializing [ 8342.318209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8342.355883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8342.362681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8348.337230] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8358.419329] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8364.590921] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8371.440169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8371.669590] Lustre: lustre-OST0000: new disk, initializing [ 8371.674795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8371.686027] Lustre: 287295:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8371.770515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.798275] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8381.998141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8382.007312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8382.118136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8386.993670] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8391.104320] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8391.281525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8395.350796] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8395.541612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8397.795373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8397.801413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8397.824096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8397.829325] Lustre: Skipped 1 previous similar message [ 8402.925351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8402.934114] Lustre: Skipped 1 previous similar message [ 8403.705854] Lustre: server umount lustre-OST0000 complete [ 8408.039176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8408.041398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8408.053072] Lustre: Skipped 2 previous similar messages [ 8413.155559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8413.159254] Lustre: Skipped 2 previous similar messages [ 8418.273491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8418.283055] Lustre: Skipped 1 previous similar message [ 8421.343434] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8421.576920] Lustre: server umount lustre-MDT0000 complete [ 8424.668460] LustreError: 287304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787448373 with bad export cookie 6256337205245208270 [ 8424.669253] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8424.679320] LustreError: 287304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8425.038786] Lustre: server umount lustre-MDT0001 complete [ 8432.180960] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:26:20 (1787448380) [ 8437.569354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.915459] LustreError: 289709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.930845] LustreError: 289709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8437.982153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8441.911601] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8443.366328] LustreError: 289710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8447.461737] LustreError: 289709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8449.654025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.958455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8453.956826] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8460.598905] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8466.160681] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8473.346764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8473.813360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8479.700025] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8487.309509] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8497.632963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8497.645270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8497.662361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8497.697207] Lustre: Skipped 1 previous similar message [ 8502.928254] Lustre: server umount lustre-OST0000 complete [ 8507.873428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.882566] Lustre: Skipped 1 previous similar message [ 8512.780437] Lustre: server umount lustre-MDT0000 complete [ 8512.999573] LustreError: 291550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8513.033427] LustreError: 291550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8516.275783] LustreError: 289688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787448465 with bad export cookie 6256337205245209152 [ 8516.292921] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8516.691764] Lustre: server umount lustre-MDT0001 complete [ 8527.852967] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8530.548434] Key type lgssc unregistered [ 8530.829674] LNet: 293248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8530.835656] LNetError: 293248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8530.852063] LNet: Removed LNI 192.168.203.155@tcp [ 8531.879145] Key type .llcrypt unregistered [ 8531.881431] Key type ._llcrypt unregistered [ 8559.494809] Key type ._llcrypt registered [ 8559.497315] Key type .llcrypt registered [ 8559.600000] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8574.482479] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8575.612351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8575.662863] alg: No test for adler32 (adler32-zlib) [ 8576.845977] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 8577.186045] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8578.879908] Key type lgssc registered [ 8579.979864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8629.586841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8629.635903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8630.978840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8631.032157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8631.173605] Lustre: lustre-MDT0000: new disk, initializing [ 8631.283777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8631.307820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8636.058595] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8647.583826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8647.701889] Lustre: 297077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8647.812526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8647.818353] Lustre: Skipped 1 previous similar message [ 8647.990103] Lustre: lustre-MDT0001: new disk, initializing [ 8648.092156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8648.135302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8648.146670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8652.845432] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8661.521738] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8666.795389] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8673.676189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8673.951941] Lustre: lustre-OST2710: new disk, initializing [ 8673.957992] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8673.965390] Lustre: 299044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8674.036794] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8680.480746] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8682.099924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8682.119471] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8682.267350] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8690.975726] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8698.930830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8699.185763] Lustre: lustre-OST03e8: new disk, initializing [ 8699.188936] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8699.195346] Lustre: 300111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8699.328141] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8706.159439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8706.180948] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8706.319121] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8706.654935] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8714.701435] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8722.178411] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8722.405665] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8726.524636] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8726.790900] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8731.501994] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8731.704217] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8736.146931] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8736.328893] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8740.324692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8740.331964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.346301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8742.375469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.375662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8742.393249] Lustre: Skipped 1 previous similar message [ 8742.399205] Lustre: Skipped 1 previous similar message [ 8746.051257] Lustre: server umount lustre-MDT0000 complete [ 8749.123691] LustreError: 297067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787448698 with bad export cookie 16046680835736962419 [ 8749.123971] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8749.138321] LustreError: 297067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8749.440257] Lustre: server umount lustre-MDT0001 complete [ 8763.014942] Lustre: server umount lustre-OST2710 complete [ 8776.906889] Lustre: server umount lustre-OST03e8 complete [ 8794.543803] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8803.530724] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8838.409431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.637362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8838.668143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8838.810714] Lustre: lustre-MDT0000: new disk, initializing [ 8838.888336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8838.900202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8843.189568] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8853.612653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8853.738665] Lustre: 305361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8853.797337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8853.801153] Lustre: Skipped 1 previous similar message [ 8853.871250] Lustre: lustre-MDT0001: new disk, initializing [ 8853.970086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8854.019543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8854.030112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8858.317312] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8867.829286] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8874.743189] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8882.041724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8882.295387] Lustre: lustre-OST0000: new disk, initializing [ 8882.310282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8882.317478] Lustre: 307331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8882.457811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8887.875072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8887.889269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8887.996305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8888.184730] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8897.144394] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8902.210674] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8902.456520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8907.244478] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8907.537259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8913.888860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8913.895900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.908191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8913.927708] Lustre: Skipped 2 previous similar messages [ 8913.956578] Lustre: Skipped 3 previous similar messages [ 8919.014056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8919.027618] Lustre: Skipped 1 previous similar message [ 8924.127394] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8924.134928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8924.150841] Lustre: Skipped 1 previous similar message [ 8924.317204] Lustre: server umount lustre-OST0000 complete [ 8929.258516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8929.275068] Lustre: Skipped 1 previous similar message [ 8933.957380] Lustre: server umount lustre-MDT0000 complete [ 8934.380349] LustreError: 305370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8934.410788] LustreError: 305370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8937.650217] LustreError: 305354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787448886 with bad export cookie 16046680835736964099 [ 8937.667236] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8938.090678] Lustre: server umount lustre-MDT0001 complete [ 8947.606805] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:34:55 (1787448895) [ 8949.399806] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8951.730630] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:34:58 (1787448898) [ 8954.932391] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8980.687028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8980.932255] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8980.945776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8981.083388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8985.714230] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8994.220931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8994.300781] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8994.332623] Lustre: 311188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8998.507566] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9020.071078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9020.281455] Lustre: Denying initial registration attempt from nid 192.168.203.155@tcp, specified as failover [ 9020.290832] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9020.297984] LustreError: 312642:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9020.303573] LustreError: 312642:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9020.308626] LustreError: 312642:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9020.378989] Lustre: server umount lustre-OST0000 complete [ 9020.385307] LustreError: 312642:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9027.554719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.558128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.568087] Lustre: Skipped 1 previous similar message [ 9027.580230] Lustre: Skipped 3 previous similar messages [ 9041.375504] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9041.599807] Lustre: server umount lustre-MDT0000 complete [ 9042.912722] LustreError: 311200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.930179] LustreError: 311200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9045.017513] LustreError: 311181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787448994 with bad export cookie 16046680835736964981 [ 9045.023341] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9045.035112] LustreError: 311181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9045.286676] Lustre: server umount lustre-MDT0001 complete [ 9056.394488] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9058.997197] Key type lgssc unregistered [ 9059.226157] LNet: 313766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9059.234905] LNetError: 313766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9059.249427] LNet: Removed LNI 192.168.203.155@tcp [ 9060.024229] Key type .llcrypt unregistered [ 9060.028085] Key type ._llcrypt unregistered [ 9070.974788] Key type ._llcrypt registered [ 9070.976978] Key type .llcrypt registered [ 9071.060896] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9072.067596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9072.082726] alg: No test for adler32 (adler32-zlib) [ 9073.091989] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 9073.266043] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9074.895187] Key type lgssc registered [ 9075.888422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9120.566719] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9130.967911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9130.979849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9132.276820] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9132.316528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9132.520652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9136.688275] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9144.774897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9144.867135] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9144.887519] Lustre: 316530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9145.180061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9149.570363] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9157.004585] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9162.883491] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9182.646563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9182.837884] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9183.043046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9188.521795] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9194.554520] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9199.584089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9199.592388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.609954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9200.096484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9200.113557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.136850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9200.141769] Lustre: Skipped 1 previous similar message [ 9202.553301] Lustre: server umount lustre-OST0000 complete [ 9206.759955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9206.767802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9206.785200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9210.337501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9210.343597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9211.854672] Lustre: server umount lustre-MDT0000 complete [ 9214.839358] LustreError: 318397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787449163 with bad export cookie 8607681603424553406 [ 9214.841817] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9214.849312] LustreError: 318397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9215.073478] Lustre: server umount lustre-MDT0001 complete [ 9223.688391] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9225.897204] Key type lgssc unregistered [ 9226.133825] LNet: 320104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9226.140407] LNetError: 320104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9226.158638] LNet: Removed LNI 192.168.203.155@tcp [ 9226.875803] Key type .llcrypt unregistered [ 9226.877647] Key type ._llcrypt unregistered [ 9247.516536] Key type ._llcrypt registered [ 9247.520120] Key type .llcrypt registered [ 9247.632723] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9248.664963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9248.689965] alg: No test for adler32 (adler32-zlib) [ 9249.905337] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 9250.223530] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9251.889759] Key type lgssc registered [ 9252.940279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9260.506779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9260.533084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9261.953867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9266.423948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9271.280470] LustreError: 321499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9271.301221] LustreError: 321499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9274.936245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.367738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9280.121803] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9288.893853] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9296.411378] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9306.216462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9306.722745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.765602] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9320.288125] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9341.923580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9341.928642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9341.942337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9341.948866] Lustre: Skipped 2 previous similar messages [ 9346.620992] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 9351.721915] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 9351.731988] Lustre: Skipped 3 previous similar messages [ 9356.255157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9356.446690] Lustre: server umount lustre-MDT0000 complete [ 9356.836473] LustreError: 321495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9359.384587] LustreError: 324107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787449308 with bad export cookie 5037902801485100903 [ 9359.391935] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9359.395348] LustreError: 324107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9361.953646] LustreError: 321494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9361.975428] LustreError: 321494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9361.988801] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 9361.995579] Lustre: Skipped 3 previous similar messages [ 9362.405094] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9362.416319] Lustre: Skipped 2 previous similar messages [ 9364.448642] LustreError: 324108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9364.464608] LustreError: 324108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9365.775702] Lustre: server umount lustre-MDT0001 complete [ 9370.971341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9378.256288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9385.329757] LustreError: 323333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9385.410215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9385.466175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9389.163547] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9394.594919] LustreError: 323333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9394.601247] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9394.616603] LustreError: 323333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9396.535344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.952053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9400.975861] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9402.358583] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9408.297150] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9409.781301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9415.036785] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9416.674249] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9421.280078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9421.284398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9421.308895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9421.321583] Lustre: Skipped 2 previous similar messages [ 9427.451542] Lustre: server umount lustre-OST0000 complete [ 9433.069221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9433.069935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9433.097215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9433.107038] Lustre: Skipped 2 previous similar messages [ 9433.131356] Lustre: Skipped 3 previous similar messages [ 9438.055602] Lustre: server umount lustre-MDT0000 complete [ 9438.176948] LustreError: 325123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9438.200289] LustreError: 325123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9441.698908] LustreError: 324311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787449390 with bad export cookie 5037902801485108071 [ 9441.704035] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9441.714943] LustreError: 324311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9441.994600] Lustre: server umount lustre-MDT0001 complete [ 9453.125700] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9455.826949] Key type lgssc unregistered [ 9456.084526] LNet: 327795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9456.093718] LNetError: 327795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9456.120067] LNet: Removed LNI 192.168.203.155@tcp [ 9457.065912] Key type .llcrypt unregistered [ 9457.071645] Key type ._llcrypt unregistered [ 9480.426210] Key type ._llcrypt registered [ 9480.429459] Key type .llcrypt registered [ 9480.538507] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9481.670094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9481.700749] alg: No test for adler32 (adler32-zlib) [ 9482.701500] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 9482.891065] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9484.527184] Key type lgssc registered [ 9485.465751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9491.915422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9491.931751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9493.273115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.588301] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9503.713232] LustreError: 329211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9503.729346] LustreError: 329211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9504.053412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.525412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9508.281782] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9514.667772] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9519.854349] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9523.173144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9523.197445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9523.203043] Lustre: Skipped 1 previous similar message [ 9528.291096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9528.298033] Lustre: Skipped 1 previous similar message [ 9528.841309] Lustre: server umount lustre-MDT0000 complete [ 9529.317151] LustreError: 330501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9529.340083] LustreError: 330501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9532.024470] LustreError: 329192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787449481 with bad export cookie 16856953052500778356 [ 9532.036017] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9532.045610] LustreError: 329192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9532.305823] Lustre: server umount lustre-MDT0001 complete [ 9545.966312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9546.281174] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9546.309512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9546.533244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9552.052111] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9561.131570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9561.321508] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9561.358673] Lustre: 331800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9561.705103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9567.054563] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9575.336571] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9581.652720] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9590.824555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.938418] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9590.946112] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9590.952474] LustreError: 333684:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9590.956418] LustreError: 333684:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9590.959691] LustreError: 333684:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9590.996434] Lustre: server umount lustre-OST0000 complete [ 9590.998670] LustreError: 333684:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9601.912967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9602.206365] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9602.405369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9608.494389] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9609.916570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9615.820596] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9623.285334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9623.371019] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9623.451193] Lustre: lustre-OST0001: new disk, initializing [ 9623.454541] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9623.464190] Lustre: 335226:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9623.544773] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9630.563910] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9632.933018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9632.944658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9633.032648] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9638.504643] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9643.504356] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9643.505904] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9643.515403] Lustre: Skipped 1 previous similar message [ 9643.521769] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9647.436759] Lustre: server umount lustre-OST0001 complete [ 9648.615700] LustreError: 334150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9648.623693] LustreError: 334150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9651.168537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9651.177344] LustreError: Skipped 1 previous similar message [ 9651.179271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.212046] Lustre: Skipped 1 previous similar message [ 9651.227050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9651.234770] Lustre: Skipped 1 previous similar message [ 9653.754021] LustreError: 334152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9653.758814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9656.992246] Lustre: server umount lustre-OST0000 complete [ 9663.968736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9663.978426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9663.979579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9663.992093] Lustre: Skipped 1 previous similar message [ 9663.999411] Lustre: Skipped 2 previous similar messages [ 9666.952601] Lustre: server umount lustre-MDT0000 complete [ 9669.093880] LustreError: 336031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9669.109251] LustreError: 336031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9670.092539] LustreError: 331793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787449619 with bad export cookie 16856953052500778902 [ 9670.100496] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9670.110635] LustreError: 331793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9670.489795] Lustre: server umount lustre-MDT0001 complete [ 9680.073680] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9682.626500] Key type lgssc unregistered [ 9682.884509] LNet: 337188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9682.892729] LNetError: 337188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9682.915278] LNet: Removed LNI 192.168.203.155@tcp [ 9683.710972] Key type .llcrypt unregistered [ 9683.713836] Key type ._llcrypt unregistered [ 9747.654962] Key type ._llcrypt registered [ 9747.656422] Key type .llcrypt registered [ 9747.742498] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9762.244675] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9763.350519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9763.377254] alg: No test for adler32 (adler32-zlib) [ 9764.513684] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 9764.809763] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9766.495164] Key type lgssc registered [ 9767.519677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9800.600409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9800.632503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9801.862846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9801.918361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9802.037738] Lustre: lustre-MDT0000: new disk, initializing [ 9802.121815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9802.158906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9806.933161] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9818.937393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9819.051240] Lustre: 342121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9819.097844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9819.101400] Lustre: Skipped 1 previous similar message [ 9819.159819] Lustre: lustre-MDT0001: new disk, initializing [ 9819.251779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9819.294731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9819.303827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9825.128802] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9835.730971] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9842.472178] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9850.296250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9850.611932] Lustre: lustre-OST0000: new disk, initializing [ 9850.614907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9850.620833] Lustre: 344090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9850.711317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9856.462651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9858.593161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9858.602374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9858.738291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9865.776507] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9871.448652] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9871.801494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9876.430166] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9876.621824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9879.008717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9879.015912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.028863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9879.525108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9879.530962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.554850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9884.646431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9884.657519] Lustre: Skipped 1 previous similar message [ 9884.912084] Lustre: server umount lustre-OST0000 complete [ 9889.760532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9889.776825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9889.789386] Lustre: Skipped 1 previous similar message [ 9894.950843] Lustre: server umount lustre-MDT0000 complete [ 9898.444684] LustreError: 342114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787449847 with bad export cookie 17198188211751920168 [ 9898.460295] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9898.469750] LustreError: 342114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9898.981343] Lustre: server umount lustre-MDT0001 complete [ 9907.742874] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:50:55 (1787449855) [ 9915.700189] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:51:02 (1787449862) [ 9924.123526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.607311] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.638431] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9924.689107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9928.623840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9929.709442] LustreError: 346848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.827150] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9936.800713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9937.159511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9941.548466] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9950.013817] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9955.987398] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9963.240976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9963.608890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9971.285844] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9981.092832] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9989.387966] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9991.218910] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9993.198717] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9995.031554] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9997.249203] Lustre: Failing over lustre-MDT0000 [ 9997.695727] Lustre: server umount lustre-MDT0000 complete [ 9997.799898] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9997.804043] LustreError: 346841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9997.827702] Lustre: Skipped 2 previous similar messages [ 9997.861368] LustreError: 346841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10002.914293] LustreError: 346848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.939294] LustreError: 346848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10013.157779] LustreError: 346841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.186034] LustreError: 346841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10017.215625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.394981] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10017.774737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10017.878730] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10020.899827] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10022.914362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10022.972799] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10023.051404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10023.301839] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10034.897696] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10036.260440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10040.045257] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10042.047322] Lustre: Failing over lustre-MDT0000 [10042.491423] Lustre: server umount lustre-MDT0000 complete [10043.365713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10043.378312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.397264] Lustre: Skipped 1 previous similar message [10043.409111] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10043.423759] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10062.362571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10062.550605] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10062.959892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10063.060850] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10066.982811] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10067.365902] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10068.478769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10068.486096] Lustre: Skipped 2 previous similar messages [10068.515193] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10068.568072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10077.342419] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10078.632428] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10088.929415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10088.937441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10088.940967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10088.941293] Lustre: Skipped 4 previous similar messages [10088.955607] Lustre: Skipped 2 previous similar messages [10090.427782] Lustre: server umount lustre-MDT0000 complete [10093.703285] LustreError: 346828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787450042 with bad export cookie 17198188211751922877 [10093.705855] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10093.711335] LustreError: 346828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10094.000191] Lustre: server umount lustre-MDT0001 complete [10107.426867] Lustre: server umount lustre-OST0000 complete [10120.555958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10121.043180] LustreError: 353187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10121.061788] LustreError: 353187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [10125.492614] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10133.262806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.706966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10133.714129] Lustre: Skipped 1 previous similar message [10137.576640] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10144.468809] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10150.499753] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10156.655055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10163.007266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10166.200359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10170.286267] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10174.445094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10174.453158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10174.477254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10180.682960] Lustre: server umount lustre-OST0000 complete [10185.186920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10185.193577] LustreError: Skipped 1 previous similar message [10191.332110] Lustre: server umount lustre-MDT0000 complete [10191.841717] LustreError: 353183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.865679] LustreError: 353183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10195.175260] LustreError: 355029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787450144 with bad export cookie 17198188211751923626 [10195.181645] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10195.192056] LustreError: 355029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10195.665495] Lustre: server umount lustre-MDT0001 complete [10206.268287] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10209.262315] Key type lgssc unregistered [10209.538559] LNet: 356721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10209.560059] LNetError: 356721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10210.604598] LNet: Removed LNI 192.168.203.155@tcp [10211.438674] Key type .llcrypt unregistered [10211.441780] Key type ._llcrypt unregistered [10233.747971] Key type ._llcrypt registered [10233.755916] Key type .llcrypt registered [10233.863986] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10235.008535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10235.062220] alg: No test for adler32 (adler32-zlib) [10236.093652] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [10236.329090] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10237.967167] Key type lgssc registered [10238.937496] Lustre: Echo OBD driver; http://www.lustre.org/ [10245.637333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10245.658766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10247.059325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10251.646502] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10257.377113] LustreError: 358220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.398916] LustreError: 358220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10259.765526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10260.107842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10264.091686] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10270.970838] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10275.965676] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10281.188553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10281.532102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10286.023226] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10292.491617] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10292.786102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10305.601622] Lustre: server umount lustre-MDT0000 complete [10306.016518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10306.035600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10306.071789] LustreError: 358220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10308.069869] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10308.071493] LustreError: 358221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10308.084093] Lustre: Skipped 1 previous similar message [10308.119598] LustreError: 358221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10310.003413] LustreError: 358201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787450258 with bad export cookie 6135187371495374188 [10310.006598] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10310.018233] LustreError: 358201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10310.300572] Lustre: server umount lustre-MDT0001 complete [10324.833573] Lustre: server umount lustre-OST0000 complete [10336.546267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10337.026243] LustreError: 361929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10337.102042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10341.401950] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10342.375238] LustreError: 361930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10348.682662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.994349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10353.190529] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10360.104222] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10366.037878] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10373.310595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10373.665780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10381.205558] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10382.924110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10389.998881] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10403.301896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.314826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10403.320229] Lustre: Skipped 1 previous similar message [10403.330388] Lustre: Skipped 2 previous similar messages [10408.420472] LustreError: 362959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10408.447225] LustreError: 362959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10408.560868] Lustre: server umount lustre-MDT0000 complete [10412.233648] LustreError: 361911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787450361 with bad export cookie 6135187371495375301 [10412.239248] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10412.242165] LustreError: 361911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10412.519431] Lustre: server umount lustre-MDT0001 complete [10426.161485] Lustre: server umount lustre-OST0000 complete [10438.952031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10439.245076] LustreError: 365840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10439.299877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10442.380806] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10448.892273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10453.428562] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10459.877990] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10465.062316] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10470.795943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10476.339755] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10482.797422] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10485.490181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10494.944886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10494.954105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.972174] Lustre: Skipped 1 previous similar message [10494.989235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10495.968843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10498.519512] Lustre: server umount lustre-OST0000 complete [10505.696659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10505.712973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10505.727498] Lustre: Skipped 1 previous similar message [10505.731584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10505.736790] Lustre: Skipped 2 previous similar messages [10508.239501] Lustre: server umount lustre-MDT0000 complete [10511.331926] LustreError: 365836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10511.345302] LustreError: 365836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10511.549985] LustreError: 365822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787450460 with bad export cookie 6135187371495376323 [10511.555243] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10511.566289] LustreError: 365822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10511.903860] Lustre: server umount lustre-MDT0001 complete [10523.046943] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10525.787480] Key type lgssc unregistered [10526.167747] LNet: 369386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10526.175219] LNetError: 369386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10526.191575] LNet: Removed LNI 192.168.203.155@tcp [10527.055699] Key type .llcrypt unregistered [10527.059945] Key type ._llcrypt unregistered [10552.656372] Key type ._llcrypt registered [10552.663977] Key type .llcrypt registered [10552.783715] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10554.330947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10554.383302] alg: No test for adler32 (adler32-zlib) [10555.426837] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [10555.665871] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10557.312165] Key type lgssc registered [10558.538562] Lustre: Echo OBD driver; http://www.lustre.org/ [10565.516802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10565.535481] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10565.550489] LustreError: 370829:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10565.571469] LustreError: 370829:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10565.580599] LustreError: 370829:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10565.586491] LustreError: 370829:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10565.594680] LustreError: 370829:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10572.731170] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10572.739103] LustreError: 371226:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10572.755910] LustreError: 371226:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10572.765588] LustreError: 371226:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10572.776234] LustreError: 371226:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10572.791195] LustreError: 371226:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10590.998822] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10593.775636] Key type lgssc unregistered [10594.079517] LNet: 372232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10594.085980] LNetError: 372232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10594.110711] LNet: Removed LNI 192.168.203.155@tcp [10594.893344] Key type .llcrypt unregistered [10594.901925] Key type ._llcrypt unregistered [10611.666743] Key type ._llcrypt registered [10611.669276] Key type .llcrypt registered [10611.782433] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10625.974066] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10627.051074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10627.082155] alg: No test for adler32 (adler32-zlib) [10628.100630] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [10628.288770] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10630.023302] Key type lgssc registered [10631.279208] Lustre: Echo OBD driver; http://www.lustre.org/ [10666.033791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10666.056093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10667.294555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10667.335189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10667.416456] Lustre: lustre-MDT0000: new disk, initializing [10667.492391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10667.513850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10671.700699] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10681.633884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.780986] Lustre: 375392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10681.831400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10681.835847] Lustre: Skipped 1 previous similar message [10681.915864] Lustre: lustre-MDT0001: new disk, initializing [10682.052632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10682.107979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10682.119462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10685.991458] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10694.706782] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10699.466320] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10705.824361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10706.006774] Lustre: lustre-OST0000: new disk, initializing [10706.011726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10706.019316] Lustre: 377358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10706.137253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10711.984349] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10712.126266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10712.143239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10712.251612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10720.139224] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10724.573897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10724.789833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10729.059664] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10729.304336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10733.028694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10733.033951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10733.043808] LustreError: Skipped 1 previous similar message [10733.060536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10733.065231] Lustre: Skipped 1 previous similar message [10738.158720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10738.164620] Lustre: Skipped 1 previous similar message [10743.277751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10743.288749] Lustre: Skipped 1 previous similar message [10745.311478] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10745.487922] Lustre: server umount lustre-OST0000 complete [10748.896558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10748.901729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10748.913244] Lustre: Skipped 1 previous similar message [10748.918302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10753.505817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10753.506146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10753.531575] Lustre: Skipped 1 previous similar message [10754.936951] Lustre: server umount lustre-MDT0000 complete [10758.078525] LustreError: 376195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787450707 with bad export cookie 16463048145124879790 [10758.083713] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10758.088700] LustreError: 376195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10758.516503] Lustre: server umount lustre-MDT0001 complete [10765.642994] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10766.927788] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:05:14 (1787450714) [10772.622748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10773.372096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10778.515394] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10778.599162] LustreError: 379817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10778.622932] LustreError: 379817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10783.720597] LustreError: 379818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.378295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10787.796471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10793.212704] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10801.686252] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10807.587461] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10814.561163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10814.903391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10821.678629] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10828.555532] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10835.047517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10835.237289] Lustre: lustre-OST0001: new disk, initializing [10835.246106] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10835.265643] Lustre: 382653:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10835.356328] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10841.256282] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10843.716116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10843.725112] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10843.783855] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10850.393400] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10858.067189] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [10859.493376] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10859.504709] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10859.512705] Lustre: Skipped 1 previous similar message [10861.076825] Lustre: server umount lustre-OST0001 complete [10863.139093] LustreError: 381639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10869.731149] LustreError: 383118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10869.749636] LustreError: 383118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10874.856917] LustreError: 381639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.877556] LustreError: 381639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10878.432829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10878.443862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10878.458748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10878.461137] Lustre: Skipped 2 previous similar messages [10883.085384] Lustre: server umount lustre-OST0000 complete [10890.208869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.224994] Lustre: Skipped 2 previous similar messages [10892.725144] Lustre: server umount lustre-MDT0000 complete [10895.329409] LustreError: 379818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.342758] LustreError: 379818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10895.375458] LustreError: 379797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787450844 with bad export cookie 16463048145124880672 [10895.378084] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10895.383486] LustreError: 379797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10901.692297] Lustre: server umount lustre-MDT0001 complete [10911.541191] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10913.985760] Key type lgssc unregistered [10914.217642] LNet: 384643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10914.226604] LNetError: 384643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10914.242558] LNet: Removed LNI 192.168.203.155@tcp [10914.968675] Key type .llcrypt unregistered [10914.972726] Key type ._llcrypt unregistered [10958.386729] Key type ._llcrypt registered [10958.388818] Key type .llcrypt registered [10958.474516] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10959.277363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10959.334223] alg: No test for adler32 (adler32-zlib) [10960.295967] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [10960.463618] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10962.111189] Key type lgssc registered [10963.098114] Lustre: Echo OBD driver; http://www.lustre.org/ [10969.350835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10969.367213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.628071] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10970.665302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10970.843256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10975.158493] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10982.956839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.067796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10983.104321] Lustre: 387420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10983.375808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10987.395779] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10994.201509] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10999.096378] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11003.873149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11003.882488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11003.901779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11004.899188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11004.899286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11010.017283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11010.024508] Lustre: Skipped 2 previous similar messages [11015.135645] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11015.138592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.151840] Lustre: Skipped 1 previous similar message [11015.377624] Lustre: server umount lustre-MDT0000 complete [11018.622213] LustreError: 388172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787450967 with bad export cookie 1893043091241712707 [11018.630364] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11018.635149] LustreError: 388172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11019.077840] Lustre: server umount lustre-MDT0001 complete [11028.284407] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:09:35 (1787450975) [11034.469266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11034.951618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11039.186624] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11040.230460] LustreError: 389933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.250892] LustreError: 389933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11045.349454] LustreError: 389934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11047.521677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11047.856409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11052.017409] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11058.871533] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11063.508542] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11068.904820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.019506] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11069.141429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11074.277058] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11080.806317] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11087.537853] Lustre: Permanently deactivating lustre-OST0000 [11087.547350] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11093.536333] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11094.726070] LustreError: 392870:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11094.733682] LustreError: 392870:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11094.740035] LustreError: 392870:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11097.584863] LustreError: 392919:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11097.592400] LustreError: 392919:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11105.520656] Lustre: server umount lustre-OST0000 complete [11107.679113] LustreError: 393168:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11107.684454] LustreError: 393168:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11109.345974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11109.351424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.360029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11111.394499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11119.628080] Lustre: server umount lustre-MDT0000 complete [11121.632556] LustreError: 389934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11121.653509] LustreError: 389934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11122.743747] LustreError: 389916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787451071 with bad export cookie 1893043091241713288 [11122.746684] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11122.754957] LustreError: 389916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11133.072890] Lustre: server umount lustre-MDT0001 complete [11141.085160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11145.217475] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11147.692450] LustreError: 394326:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11153.827618] Lustre: 394664:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.155@tcp [11155.647694] Lustre: server umount MGS complete [11166.644530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.991503] LustreError: 395388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11167.025700] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11167.028027] Lustre: Skipped 1 previous similar message [11167.083213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11171.541468] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11172.327895] LustreError: 395388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11172.351270] LustreError: 395388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11179.411046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11179.711748] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11184.757074] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11192.030933] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11198.124882] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11200.017490] Lustre: Permanently reactivating lustre-OST0000 [11200.027587] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11200.035728] Lustre: Skipped 2 previous similar messages [11226.591752] LustreError: 397144:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11233.997381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11234.391499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11234.396461] Lustre: Skipped 1 previous similar message [11239.852461] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11246.377319] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11252.194407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11252.198790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11252.200064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11252.209379] Lustre: Skipped 1 previous similar message [11252.219335] Lustre: Skipped 5 previous similar messages [11256.577864] Lustre: server umount lustre-OST0000 complete [11261.922927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11261.937342] LustreError: Skipped 1 previous similar message [11261.945451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11266.350669] Lustre: server umount lustre-MDT0000 complete [11267.555441] LustreError: 395388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11267.565907] LustreError: 395388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11269.692625] LustreError: 399049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787451218 with bad export cookie 1893043091241714345 [11269.698296] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11269.712167] LustreError: 399049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11269.943296] Lustre: server umount lustre-MDT0001 complete [11280.314803] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11283.273934] Key type lgssc unregistered [11283.625765] LNet: 399804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11283.642805] LNetError: 399804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11283.659652] LNet: Removed LNI 192.168.203.155@tcp [11284.503715] Key type .llcrypt unregistered [11284.506689] Key type ._llcrypt unregistered [11303.957647] Key type ._llcrypt registered [11303.960619] Key type .llcrypt registered [11304.073846] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [11317.616177] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11318.362586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11318.400039] alg: No test for adler32 (adler32-zlib) [11319.483859] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [11319.676512] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11321.343264] Key type lgssc registered [11322.488952] Lustre: Echo OBD driver; http://www.lustre.org/ [11357.005802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11357.030165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11358.276396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11358.318654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11358.403809] Lustre: lustre-MDT0000: new disk, initializing [11358.464835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11358.486060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11362.095147] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11370.893810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11371.043299] Lustre: 402963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11371.115545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11371.121666] Lustre: Skipped 1 previous similar message [11371.216927] Lustre: lustre-MDT0001: new disk, initializing [11371.315782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11371.370379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11371.379685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11375.216313] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11382.807767] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11387.951480] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11393.683724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11394.006749] Lustre: lustre-OST0000: new disk, initializing [11394.013090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11394.022276] Lustre: 404929:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11394.103080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11400.211659] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11400.243344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11400.255435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11400.360239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11408.673491] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11412.702981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11412.923945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11417.894432] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11418.145296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11421.158367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11421.160054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11421.206847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11426.106480] Lustre: server umount lustre-OST0000 complete [11431.393407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11431.413884] Lustre: Skipped 1 previous similar message [11431.424114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11431.433183] Lustre: Skipped 2 previous similar messages [11435.347712] Lustre: server umount lustre-MDT0000 complete [11438.516969] LustreError: 402954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787451387 with bad export cookie 11851218159252646178 [11438.522024] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11438.528112] LustreError: 402954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11438.872629] Lustre: server umount lustre-MDT0001 complete [11447.170907] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11448.905638] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:16:36 (1787451396) [11455.498293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11455.915201] LustreError: 407389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11455.946762] LustreError: 407389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11456.058344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11460.063474] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11461.090451] LustreError: 407390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11461.099658] LustreError: 407390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11465.206910] LustreError: 407389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11468.053778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11468.415188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11472.981234] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11481.077181] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11486.880660] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11493.353338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11493.700762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11499.408826] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11507.403029] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11509.681220] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11512.414732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11520.483715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11520.495916] LustreError: Skipped 1 previous similar message [11520.505087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11520.519866] Lustre: Skipped 1 previous similar message [11520.527777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11521.516083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11523.653779] Lustre: server umount lustre-OST0000 complete [11529.697907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11529.711506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11529.738584] Lustre: Skipped 1 previous similar message [11529.751112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11529.763865] Lustre: Skipped 2 previous similar messages [11533.954354] Lustre: server umount lustre-MDT0000 complete [11536.864475] LustreError: 408117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11536.883165] LustreError: 408117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11537.426978] LustreError: 407370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787451486 with bad export cookie 11851218159252647060 [11537.430233] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11537.438525] LustreError: 407370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11537.754414] Lustre: server umount lustre-MDT0001 complete [11548.025015] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11550.544753] Key type lgssc unregistered [11550.856837] LNet: 411117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11550.868332] LNetError: 411117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11550.883929] LNet: Removed LNI 192.168.203.155@tcp [11551.680839] Key type .llcrypt unregistered [11551.682935] Key type ._llcrypt unregistered [11573.640456] Key type ._llcrypt registered [11573.644349] Key type .llcrypt registered [11573.753325] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11574.036252] Key type .llcrypt unregistered [11574.039299] Key type ._llcrypt unregistered [11590.051575] Key type ._llcrypt registered [11590.053185] Key type .llcrypt registered [11590.124094] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11590.624794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11590.661235] alg: No test for adler32 (adler32-zlib) [11591.891769] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [11592.236521] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11593.919179] Key type lgssc registered [11595.212672] Lustre: Echo OBD driver; http://www.lustre.org/ [11605.328881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11605.370319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11607.027671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11612.249383] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11616.226461] LustreError: 413334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11616.259353] LustreError: 413334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11620.869817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11621.354298] LustreError: 413335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11621.460865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11626.477775] LustreError: 413331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11626.501891] LustreError: 413331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11628.666643] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11631.586758] LustreError: 413335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11631.602897] LustreError: 413335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11631.605156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11636.707312] LustreError: 413330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11636.733980] LustreError: 413330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11637.508253] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11644.447360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11644.827303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11649.236611] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11660.013495] Lustre: server umount lustre-OST0000 complete [11665.385785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11665.394099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11665.415034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11670.497534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11670.503355] Lustre: Skipped 2 previous similar messages [11674.658319] Lustre: server umount lustre-MDT0000 complete [11675.617715] LustreError: 413334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.642541] LustreError: 413334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11676.640642] Lustre: 412646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787451609/real 1787451609] req@0000000097ebe381 x1874278787082240/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787451625 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11676.670703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11676.689362] Lustre: Skipped 1 previous similar message [11678.533471] LustreError: 415081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787451627 with bad export cookie 2533357611455789332 [11678.543876] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11678.544280] LustreError: 415081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11678.928259] Lustre: server umount lustre-MDT0001 complete [11691.162874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11693.834529] Key type lgssc unregistered [11694.182519] LNet: 416489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11694.189570] LNetError: 416489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11694.208302] LNet: Removed LNI 192.168.203.155@tcp [11695.047106] Key type .llcrypt unregistered [11695.049309] Key type ._llcrypt unregistered [11721.661524] Key type ._llcrypt registered [11721.668953] Key type .llcrypt registered [11721.773682] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11723.131490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11723.181113] alg: No test for adler32 (adler32-zlib) [11724.183455] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [11724.403541] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11726.089341] Key type lgssc registered [11727.694114] Lustre: Echo OBD driver; http://www.lustre.org/ [11735.351308] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11735.367638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11750.943512] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11756.063317] LustreError: 417878:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [11782.690060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11788.949669] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11796.340821] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11803.029688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11803.534870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11803.622715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11807.911442] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11808.741026] LustreError: 417904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11808.764946] LustreError: 417904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11813.872504] LustreError: 417903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11815.770138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11815.964805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11820.954221] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11829.818256] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11836.033710] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11843.553861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11843.565995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11843.591947] Lustre: Skipped 1 previous similar message [11844.497781] Lustre: server umount lustre-OST0000 complete [11848.685729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.690146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11848.699929] Lustre: Skipped 1 previous similar message [11848.703303] Lustre: Skipped 1 previous similar message [11852.763831] Lustre: server umount lustre-MDT0000 complete [11853.792594] LustreError: 418919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.827767] LustreError: 418919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11856.073128] LustreError: 417886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787451805 with bad export cookie 4520875809504786038 [11856.079363] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11856.088599] LustreError: 417886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11856.389746] Lustre: server umount lustre-MDT0001 complete [11866.604875] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11869.126303] Key type lgssc unregistered [11869.414501] LNet: 421450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11869.423371] LNetError: 421450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11869.458033] LNet: Removed LNI 192.168.203.155@tcp [11870.163773] Key type .llcrypt unregistered [11870.165549] Key type ._llcrypt unregistered [11893.026548] Key type ._llcrypt registered [11893.029977] Key type .llcrypt registered [11893.150270] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11893.776379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11893.794337] alg: No test for adler32 (adler32-zlib) [11894.871546] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [11895.061775] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11896.720088] Key type lgssc registered [11897.678359] Lustre: Echo OBD driver; http://www.lustre.org/ [11904.265091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11904.291873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11905.656877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11909.989567] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11915.242959] LustreError: 422865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.264849] LustreError: 422865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11918.435649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.804832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11923.409697] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11930.637851] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11936.322587] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11942.382763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11942.666228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11948.355608] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11950.901489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11956.470735] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11961.320069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11961.322626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11961.341720] Lustre: Skipped 2 previous similar messages [11961.350703] Lustre: Skipped 1 previous similar message [11964.506634] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [11964.512832] Lustre: Skipped 1 previous similar message [11966.435056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11966.443850] Lustre: Skipped 2 previous similar messages [11967.574758] Lustre: server umount lustre-MDT0000 complete [11969.621351] LustreError: 422845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.55@tcp arrived at 1787451918 with bad export cookie 6680876388069148662 [11969.635122] LustreError: 422845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11971.172843] LustreError: 422846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787451920 with bad export cookie 6680876388069147766 [11971.185506] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11971.186991] LustreError: 422846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11971.554824] LustreError: 422866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11971.555324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11971.573189] LustreError: 422866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11971.601060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11976.676451] LustreError: 422865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11976.677241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11976.697686] LustreError: 422865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11977.402761] Lustre: server umount lustre-OST0000 complete [11981.798698] LustreError: 425445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11981.807514] LustreError: 425445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11983.892738] Lustre: server umount lustre-MDT0001 complete [11995.497507] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11999.844833] Key type lgssc unregistered [12000.166722] LNet: 426553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12000.171820] LNetError: 426553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12000.196669] LNet: Removed LNI 192.168.203.155@tcp [12001.613453] Key type .llcrypt unregistered [12001.621503] Key type ._llcrypt unregistered [12030.263841] Key type ._llcrypt registered [12030.270552] Key type .llcrypt registered [12030.434164] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12031.994437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12032.066809] alg: No test for adler32 (adler32-zlib) [12033.324860] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [12033.640767] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12035.455180] Key type lgssc registered [12036.732330] Lustre: Echo OBD driver; http://www.lustre.org/ [12046.278434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12046.293880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12047.792593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12054.006857] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12057.059365] LustreError: 427975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12057.091693] LustreError: 427975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12062.196652] LustreError: 427976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12063.571428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12063.944844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12068.767729] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12076.852944] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12082.250647] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12088.494879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12088.740525] LustreError: 429798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12088.952661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12093.040827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12095.536636] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12103.028199] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12108.773865] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12108.785377] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12108.797611] Lustre: Skipped 1 previous similar message [12108.824412] Lustre: Skipped 1 previous similar message [12109.897396] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [12112.683727] Lustre: server umount lustre-MDT0001 complete [12113.891127] LustreError: 427970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12113.902547] LustreError: 427970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12119.010448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12119.019169] LustreError: 428980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12119.049729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.083979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12124.130185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12124.637411] Lustre: server umount lustre-OST0000 complete [12128.588604] Lustre: server umount lustre-MDT0000 complete [12142.419563] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12145.204743] Key type lgssc unregistered [12145.564577] LNet: 431665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12145.588356] LNetError: 431665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12145.616862] LNet: Removed LNI 192.168.203.155@tcp [12146.505263] Key type .llcrypt unregistered [12146.510309] Key type ._llcrypt unregistered [12171.293686] Key type ._llcrypt registered [12171.296354] Key type .llcrypt registered [12171.422828] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12186.090517] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12187.614707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12187.651052] alg: No test for adler32 (adler32-zlib) [12188.790440] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [12189.034742] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12190.711190] Key type lgssc registered [12191.667201] Lustre: Echo OBD driver; http://www.lustre.org/ [12226.688693] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12241.646423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12241.695137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12242.950725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12242.988366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12243.053677] Lustre: lustre-MDT0000: new disk, initializing [12243.196483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12243.223613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12247.332720] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12263.493490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12263.729275] Lustre: lustre-OST0000: new disk, initializing [12263.733365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12263.736604] Lustre: Skipped 1 previous similar message [12263.741414] Lustre: 436701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12263.818532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12270.222836] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12272.249593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12272.260071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12272.287820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12279.957123] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12295.180922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12295.304640] Lustre: 435681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12295.376035] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12295.447194] Lustre: lustre-MDT0001: new disk, initializing [12295.582954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12295.640904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12295.654961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12299.860092] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12309.064981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12309.357196] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12313.757504] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12313.978520] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12319.204075] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12319.214781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12319.215934] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12319.227091] Lustre: Skipped 1 previous similar message [12319.824748] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [12319.832384] Lustre: Skipped 1 previous similar message [12324.324032] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12324.335867] Lustre: Skipped 1 previous similar message [12325.040827] Lustre: server umount lustre-MDT0001 complete [12334.573871] LustreError: 435688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12334.610380] LustreError: 435688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12339.681305] LustreError: 439272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12339.725617] LustreError: 439272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12344.293112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12344.296077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12344.317368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12344.321674] Lustre: Skipped 1 previous similar message [12344.818405] LustreError: 435694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12348.127586] Lustre: server umount lustre-OST0000 complete [12349.945128] LustreError: 439272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.001330] Lustre: server umount lustre-MDT0000 complete [12372.949695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12375.863346] Key type lgssc unregistered [12376.266700] LNet: 440455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12376.291586] LNetError: 440455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12376.316724] LNet: Removed LNI 192.168.203.155@tcp [12377.832660] Key type .llcrypt unregistered [12377.841080] Key type ._llcrypt unregistered [12419.451481] Key type ._llcrypt registered [12419.454827] Key type .llcrypt registered [12419.564719] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12419.849910] Key type .llcrypt unregistered [12419.854568] Key type ._llcrypt unregistered [12436.216062] Key type ._llcrypt registered [12436.217679] Key type .llcrypt registered [12436.331379] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12437.376473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12437.395075] alg: No test for adler32 (adler32-zlib) [12438.447538] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [12438.758625] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12440.423292] Key type lgssc registered [12441.600959] Lustre: Echo OBD driver; http://www.lustre.org/ [12466.381272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12466.424418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12467.634154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12467.675433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12467.771252] Lustre: lustre-MDT0000: new disk, initializing [12467.871163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12467.900780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12472.187249] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12482.092077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.213251] Lustre: 443835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12482.288468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12482.294869] Lustre: Skipped 1 previous similar message [12482.378468] Lustre: lustre-MDT0001: new disk, initializing [12482.493703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12482.530124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12482.538166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12486.449809] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12496.812909] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12502.946759] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12510.376487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12510.857078] Lustre: lustre-OST0000: new disk, initializing [12510.863950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12510.880815] Lustre: 445803:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12511.011681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12518.611329] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12520.528834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12520.550071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12520.675527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12528.392043] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12536.296557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12536.304584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12536.309234] Lustre: Skipped 2 previous similar messages [12536.320112] Lustre: Skipped 1 previous similar message [12541.181061] Lustre: server umount lustre-MDT0000 complete [12544.440974] LustreError: 444638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787452493 with bad export cookie 6627263346509983482 [12544.444946] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12544.451307] LustreError: 444638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12544.811762] Lustre: server umount lustre-MDT0001 complete [12558.752625] Lustre: server umount lustre-OST0000 complete [12570.351322] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:35:17 (1787452517) [12579.044919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.558062] LustreError: 447829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12579.578048] LustreError: 447829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12579.623474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12583.586017] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12584.931921] LustreError: 447830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12589.029615] LustreError: 447829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12592.170511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12592.554535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12597.231350] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12604.673671] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12609.969275] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12630.078398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12630.512463] Lustre: Found index 0 for lustre-OST0000, updating log [12630.746948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12637.741027] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12645.799798] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12650.282896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12658.114784] Lustre: server umount lustre-OST0000 complete [12674.527976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12674.544697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12674.582055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12675.615886] Lustre: 442370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787452607/real 1787452607] req@000000003a55f419 x1874279674223872/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787452623 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12675.650765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12675.678080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12675.683330] Lustre: Skipped 1 previous similar message [12676.703216] Lustre: 442370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787452609/real 1787452609] req@0000000003f44dd7 x1874279674224896/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787452625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12681.184317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12681.196878] Lustre: Skipped 1 previous similar message [12682.207342] Lustre: 442370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787452614/real 1787452614] req@00000000b91ed112 x1874279674226048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787452630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12683.334299] Lustre: server umount lustre-MDT0000 complete [12685.281437] LustreError: 447830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12685.300523] LustreError: 447830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12686.306433] Lustre: 442367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787452619/real 1787452619] req@000000006672aa00 x1874279674227200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787452635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12687.053714] LustreError: 447812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787452636 with bad export cookie 6627263346509984875 [12687.058267] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12687.076918] LustreError: 447812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12687.427487] Lustre: server umount lustre-MDT0001 complete [12689.632124] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [12696.696583] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:37:24 (1787452644) [13315.131556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13315.651663] LustreError: 454242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13315.681182] LustreError: 454242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13315.798371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13320.528801] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13321.189272] LustreError: 454243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13325.287809] LustreError: 454242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13329.302585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13329.736236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13334.505667] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13344.274482] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13351.049908] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13367.519258] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [13372.784844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13373.103360] Lustre: Found index 0 for lustre-OST0000, updating log [13373.555398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13379.835354] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13388.239499] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13391.591033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13391.616063] Lustre: Skipped 1 previous similar message [13391.627817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13396.837821] Lustre: server umount lustre-OST0000 complete [13401.569210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13401.582148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13401.619594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13402.658463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13402.661909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13404.769872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13404.783558] Lustre: Skipped 4 previous similar messages [13407.710730] Lustre: server umount lustre-MDT0000 complete [13407.844871] LustreError: 454242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13407.863496] LustreError: 454242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13411.553885] LustreError: 454224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787453360 with bad export cookie 6627263346509985960 [13411.565945] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13411.571389] LustreError: 454224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13411.957788] Lustre: server umount lustre-MDT0001 complete [13427.935327] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [13442.741309] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13453.686638] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13493.125395] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:50:40 (1787453440) [13513.741932] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13517.192775] Key type lgssc unregistered [13517.615495] LNet: 460855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13517.626656] LNetError: 460855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13633.382930] LNet: Removed LNI 192.168.203.155@tcp [13634.356897] Key type .llcrypt unregistered [13634.363122] Key type ._llcrypt unregistered [13651.791588] Key type ._llcrypt registered [13651.794386] Key type .llcrypt registered [13651.867697] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13652.624202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13652.686773] alg: No test for adler32 (adler32-zlib) [13653.630751] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [13653.797104] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13655.407157] Key type lgssc registered [13656.767880] Lustre: Echo OBD driver; http://www.lustre.org/ [13661.890388] LNet: Added LNI 192.168.203.155@tcp100 [8/256/0/180] [13662.891092] LNet: Added LNI 192.168.203.155@tcp101 [8/256/0/180] [13663.877955] LNet: Added LNI 192.168.203.155@tcp102 [8/256/0/180] [13664.928842] LNet: Added LNI 192.168.203.155@tcp103 [8/256/0/180] [13665.913467] LNet: Added LNI 192.168.203.155@tcp104 [8/256/0/180] [13667.042428] LNet: Added LNI 192.168.203.155@tcp105 [8/256/0/180] [13667.982359] LNet: Added LNI 192.168.203.155@tcp106 [8/256/0/180] [13669.058805] LNet: Added LNI 192.168.203.155@tcp107 [8/256/0/180] [13669.920421] LNet: Added LNI 192.168.203.155@tcp108 [8/256/0/180] [13670.870250] LNet: Added LNI 192.168.203.155@tcp109 [8/256/0/180] [13671.755355] LNet: Added LNI 192.168.203.155@tcp110 [8/256/0/180] [13672.617319] LNet: Added LNI 192.168.203.155@tcp111 [8/256/0/180] [13673.683736] LNet: Added LNI 192.168.203.155@tcp112 [8/256/0/180] [13674.823873] LNet: Added LNI 192.168.203.155@tcp113 [8/256/0/180] [13675.860857] LNet: Added LNI 192.168.203.155@tcp114 [8/256/0/180] [13676.835256] LNet: Added LNI 192.168.203.155@tcp115 [8/256/0/180] [13677.774837] LNet: Added LNI 192.168.203.155@tcp116 [8/256/0/180] [13678.827871] LNet: Added LNI 192.168.203.155@tcp117 [8/256/0/180] [13679.786393] LNet: Added LNI 192.168.203.155@tcp118 [8/256/0/180] [13681.091618] LNet: Added LNI 192.168.203.155@tcp119 [8/256/0/180] [13681.908927] LNet: Added LNI 192.168.203.155@tcp120 [8/256/0/180] [13682.795189] LNet: Added LNI 192.168.203.155@tcp121 [8/256/0/180] [13683.753477] LNet: Added LNI 192.168.203.155@tcp122 [8/256/0/180] [13684.695167] LNet: Added LNI 192.168.203.155@tcp123 [8/256/0/180] [13685.783891] LNet: Added LNI 192.168.203.155@tcp124 [8/256/0/180] [13686.946869] LNet: Added LNI 192.168.203.155@tcp125 [8/256/0/180] [13687.940097] LNet: Added LNI 192.168.203.155@tcp126 [8/256/0/180] [13688.929402] LNet: Added LNI 192.168.203.155@tcp127 [8/256/0/180] [13689.932571] LNet: Added LNI 192.168.203.155@tcp128 [8/256/0/180] [13691.153753] LNet: Added LNI 192.168.203.155@tcp129 [8/256/0/180] [13692.064575] LNet: Added LNI 192.168.203.155@tcp130 [8/256/0/180] [13692.951726] LNet: Added LNI 192.168.203.155@tcp131 [8/256/0/180] [13693.924572] LNet: Added LNI 192.168.203.155@tcp132 [8/256/0/180] [13694.766426] LNet: Added LNI 192.168.203.155@tcp133 [8/256/0/180] [13695.829029] LNet: Added LNI 192.168.203.155@tcp134 [8/256/0/180] [13696.761197] LNet: Added LNI 192.168.203.155@tcp135 [8/256/0/180] [13704.616038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13704.637938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13705.808187] Lustre: 463887:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13705.888526] LNetError: 463907:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13705.899858] LNetError: Protocol error connecting to 192.168.203.155@tcp120 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13705.909406] LNetError: Skipped 3 previous similar messages [13706.015280] LNet: 10 local NIs in recovery (showing 10): 192.168.203.155@tcp121, 192.168.203.155@tcp122, 192.168.203.155@tcp123, 192.168.203.155@tcp125, 192.168.203.155@tcp126, 192.168.203.155@tcp128, 192.168.203.155@tcp129, 192.168.203.155@tcp127, 192.168.203.155@tcp124, 192.168.203.155@tcp120 [13706.032638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13706.096711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13706.214956] Lustre: lustre-MDT0000: new disk, initializing [13706.325588] Lustre: 463943:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13706.335403] Lustre: 463943:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13706.354097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13706.373908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13708.144395] LNetError: 461348:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13708.161861] LNetError: 461348:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 24 previous similar messages [13708.177199] LNetError: Protocol error connecting to 192.168.203.155@tcp122 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13708.187644] LNetError: Skipped 16 previous similar messages [13711.211613] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13712.228134] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13712.257402] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [13712.270665] LNetError: Protocol error connecting to 192.168.203.155@tcp124 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13712.289571] LNetError: Skipped 1 previous similar message [13720.416452] LNetError: 464022:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13720.426918] LNetError: Protocol error connecting to 192.168.203.155@tcp124 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13722.117122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13722.258809] Lustre: 463904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13722.386898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13722.391582] Lustre: Skipped 1 previous similar message [13722.539334] Lustre: lustre-MDT0001: new disk, initializing [13722.608817] Lustre: 464707:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13722.638448] Lustre: 464707:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13722.783807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13722.846051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13722.871288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13727.367705] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13730.822273] Lustre: 465091:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13730.833216] Lustre: 465091:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13735.712671] LNetError: 461348:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13735.732584] LNetError: Protocol error connecting to 192.168.203.155@tcp108 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13738.013526] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13743.942837] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13744.932383] LNetError: 461346:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13744.948204] LNetError: 461346:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [13744.959670] LNetError: Protocol error connecting to 192.168.203.155@tcp128 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13744.985523] LNetError: Skipped 6 previous similar messages [13750.688333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13751.035813] Lustre: lustre-OST0000: new disk, initializing [13751.042386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13751.055615] Lustre: 465879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13751.152841] Lustre: 465884:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13751.160415] Lustre: 465884:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13751.230453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13757.071879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13760.043480] Lustre: 466337:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13760.053967] Lustre: 466337:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13760.133298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13760.149863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13760.317288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13762.346721] LNetError: 463914:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13762.359087] LNetError: 463914:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [13762.378100] LNetError: Protocol error connecting to 192.168.203.155@tcp129 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13762.390755] LNetError: Skipped 13 previous similar messages [13766.515356] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp124 [13767.206193] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13775.842690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13775.843220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13775.848580] Lustre: Skipped 1 previous similar message [13775.851921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13775.852747] LustreError: Skipped 1 previous similar message [13780.973575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13780.979447] Lustre: Skipped 1 previous similar message [13781.462769] Lustre: server umount lustre-OST0000 complete [13786.087824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13786.107445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13786.115023] Lustre: Skipped 2 previous similar messages [13791.202901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13791.216342] Lustre: Skipped 1 previous similar message [13791.654401] Lustre: server umount lustre-MDT0000 complete [13794.823219] LustreError: 463897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787453743 with bad export cookie 17086892209463423147 [13794.824880] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13794.837027] LustreError: 463897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13795.198865] Lustre: server umount lustre-MDT0001 complete [13801.249457] LNetError: 463918:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13801.262884] LNetError: 463918:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [13801.273960] LNetError: Protocol error connecting to 192.168.203.155@tcp119 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13801.287082] LNetError: Skipped 18 previous similar messages [13804.823764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13807.161803] Key type lgssc unregistered [13807.484436] LNet: 467794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13807.498964] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13807.507627] LNet: Removed LNI 192.168.203.155@tcp [13808.543130] LNet: Removed LNI 192.168.203.155@tcp100 [13808.555262] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13808.563412] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13808.588321] LNet: Removed LNI 192.168.203.155@tcp101 [13809.632397] LNet: Removed LNI 192.168.203.155@tcp102 [13809.637819] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13809.651020] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13809.663739] LNet: Removed LNI 192.168.203.155@tcp103 [13810.719115] LNet: Removed LNI 192.168.203.155@tcp104 [13811.743412] LNet: Removed LNI 192.168.203.155@tcp105 [13811.755063] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13811.775641] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13811.783219] LNet: Removed LNI 192.168.203.155@tcp106 [13812.832863] LNet: Removed LNI 192.168.203.155@tcp107 [13813.855167] LNet: Removed LNI 192.168.203.155@tcp108 [13814.879395] LNet: Removed LNI 192.168.203.155@tcp109 [13815.904621] LNet: Removed LNI 192.168.203.155@tcp110 [13815.910257] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13815.916619] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13815.928800] LNet: Removed LNI 192.168.203.155@tcp111 [13816.991206] LNet: Removed LNI 192.168.203.155@tcp112 [13818.016565] LNet: Removed LNI 192.168.203.155@tcp113 [13819.039548] LNet: Removed LNI 192.168.203.155@tcp114 [13820.063194] LNet: Removed LNI 192.168.203.155@tcp115 [13821.088068] LNet: Removed LNI 192.168.203.155@tcp116 [13822.111160] LNet: Removed LNI 192.168.203.155@tcp117 [13823.135173] LNet: Removed LNI 192.168.203.155@tcp118 [13824.159252] LNet: Removed LNI 192.168.203.155@tcp119 [13824.165339] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13824.177155] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13824.191307] LNet: Removed LNI 192.168.203.155@tcp120 [13825.247208] LNet: Removed LNI 192.168.203.155@tcp121 [13826.271314] LNet: Removed LNI 192.168.203.155@tcp122 [13827.295269] LNet: Removed LNI 192.168.203.155@tcp123 [13827.297460] LNet: Removed LNI 192.168.203.155@tcp124 [13828.319157] LNet: Removed LNI 192.168.203.155@tcp125 [13829.343342] LNet: Removed LNI 192.168.203.155@tcp126 [13830.367236] LNet: Removed LNI 192.168.203.155@tcp127 [13831.459324] LNet: Removed LNI 192.168.203.155@tcp128 [13832.479151] LNet: Removed LNI 192.168.203.155@tcp129 [13832.483592] LNet: Removed LNI 192.168.203.155@tcp130 [13832.488044] LNet: Removed LNI 192.168.203.155@tcp131 [13832.492283] LNet: Removed LNI 192.168.203.155@tcp132 [13832.497257] LNet: Removed LNI 192.168.203.155@tcp133 [13832.505829] LNet: Removed LNI 192.168.203.155@tcp134 [13832.529667] LNet: Removed LNI 192.168.203.155@tcp135 [13833.685752] Key type .llcrypt unregistered [13833.688264] Key type ._llcrypt unregistered [13856.048961] Key type ._llcrypt registered [13856.057114] Key type .llcrypt registered [13856.305787] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13870.791873] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13872.076194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13872.130286] alg: No test for adler32 (adler32-zlib) [13873.209417] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [13873.402309] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13875.063162] Key type lgssc registered [13876.173786] Lustre: Echo OBD driver; http://www.lustre.org/ [13909.686456] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:57:37 (1787453857) [13927.881953] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13930.673913] Key type lgssc unregistered [13930.966330] LNet: 471884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13930.977567] LNetError: 471884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13930.997338] LNet: Removed LNI 192.168.203.155@tcp [13932.034574] Key type .llcrypt unregistered [13932.038000] Key type ._llcrypt unregistered [13948.870761] Key type ._llcrypt registered [13948.873479] Key type .llcrypt registered [13948.988962] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13950.052670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13950.130540] alg: No test for adler32 (adler32-zlib) [13951.221427] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [13951.619529] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13953.367324] Key type lgssc registered [13954.419601] Lustre: Echo OBD driver; http://www.lustre.org/ [13964.192266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13964.226283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13965.561944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13965.592371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13965.688656] Lustre: lustre-MDT0000: new disk, initializing [13965.754317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13965.768454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13970.774946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13981.781241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13981.868403] Lustre: 473100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13981.925839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13981.930762] Lustre: Skipped 1 previous similar message [13982.019819] Lustre: lustre-MDT0001: new disk, initializing [13982.142586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13982.181551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13982.199577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13987.157685] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13997.693409] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14003.613611] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14011.852137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14012.240421] Lustre: lustre-OST0000: new disk, initializing [14012.247555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14012.257792] Lustre: 475068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14012.377977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14020.319567] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14021.787388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14021.800063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14021.960853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14030.816440] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14034.440635] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lnet_if_list [14040.318564] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14040.351131] LNet: Added LNI 192.168.203.155@tcp10 [8/256/0/180] [14049.259311] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14049.328538] LNet: Added LNI 192.168.203.155@tcp11 [8/256/0/180] [14055.328451] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14055.379265] LNet: Added LNI 192.168.203.155@tcp12 [8/256/0/180] [14060.881626] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14060.903396] LNet: Added LNI 192.168.203.155@tcp13 [8/256/0/180] [14066.438734] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14066.481602] LNet: Added LNI 192.168.203.155@tcp14 [8/256/0/180] [14071.837428] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14071.888803] LNet: Added LNI 192.168.203.155@tcp15 [8/256/0/180] [14077.531760] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14077.587956] LNet: Added LNI 192.168.203.155@tcp16 [8/256/0/180] [14083.805797] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14083.856198] LNet: Added LNI 192.168.203.155@tcp17 [8/256/0/180] [14089.353361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14089.406689] LNet: Added LNI 192.168.203.155@tcp18 [8/256/0/180] [14094.858913] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14094.898507] LNet: Added LNI 192.168.203.155@tcp19 [8/256/0/180] [14100.951925] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14100.990298] LNet: Added LNI 192.168.203.155@tcp20 [8/256/0/180] [14106.973344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14107.013337] LNet: Added LNI 192.168.203.155@tcp21 [8/256/0/180] [14113.096817] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14113.131356] LNet: Added LNI 192.168.203.155@tcp22 [8/256/0/180] [14118.944249] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14118.994219] LNet: Added LNI 192.168.203.155@tcp23 [8/256/0/180] [14124.880796] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14124.906658] LNet: Added LNI 192.168.203.155@tcp24 [8/256/0/180] [14130.170751] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14130.195834] LNet: Added LNI 192.168.203.155@tcp25 [8/256/0/180] [14137.186828] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14137.216845] LNet: Added LNI 192.168.203.155@tcp26 [8/256/0/180] [14143.805699] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14143.827556] LNet: Added LNI 192.168.203.155@tcp27 [8/256/0/180] [14150.786469] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14150.833668] LNet: Added LNI 192.168.203.155@tcp28 [8/256/0/180] [14158.145907] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14158.171916] LNet: Added LNI 192.168.203.155@tcp29 [8/256/0/180] [14164.609448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14164.628776] LNet: Added LNI 192.168.203.155@tcp30 [8/256/0/180] [14171.137974] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14171.173942] LNet: Added LNI 192.168.203.155@tcp31 [8/256/0/180] [14178.410674] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14178.455034] LNet: Added LNI 192.168.203.155@tcp32 [8/256/0/180] [14185.571858] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14185.606541] LNet: Added LNI 192.168.203.155@tcp33 [8/256/0/180] [14192.092669] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14192.114695] LNet: Added LNI 192.168.203.155@tcp34 [8/256/0/180] [14198.673327] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14198.700780] LNet: Added LNI 192.168.203.155@tcp35 [8/256/0/180] [14205.356378] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14205.401415] LNet: Added LNI 192.168.203.155@tcp36 [8/256/0/180] [14212.135679] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14212.167227] LNet: Added LNI 192.168.203.155@tcp37 [8/256/0/180] [14218.593204] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14218.631453] LNet: Added LNI 192.168.203.155@tcp38 [8/256/0/180] [14225.346216] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14225.370196] LNet: Added LNI 192.168.203.155@tcp39 [8/256/0/180] [14231.975449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14232.008623] LNet: Added LNI 192.168.203.155@tcp40 [8/256/0/180] [14239.290584] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14239.347857] LNet: Added LNI 192.168.203.155@tcp41 [8/256/0/180] [14246.427774] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14246.458183] LNet: Added LNI 192.168.203.155@tcp42 [8/256/0/180] [14253.058188] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14253.083971] LNet: Added LNI 192.168.203.155@tcp43 [8/256/0/180] [14260.631859] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14260.671456] LNet: Added LNI 192.168.203.155@tcp44 [8/256/0/180] [14269.064862] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14269.100067] LNet: Added LNI 192.168.203.155@tcp45 [8/256/0/180] [14279.054905] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14279.089937] LNet: Added LNI 192.168.203.155@tcp46 [8/256/0/180] [14286.681123] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14286.705822] LNet: Added LNI 192.168.203.155@tcp47 [8/256/0/180] [14295.076048] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14295.110565] LNet: Added LNI 192.168.203.155@tcp48 [8/256/0/180] [14303.656126] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14303.683721] LNet: Added LNI 192.168.203.155@tcp49 [8/256/0/180] [14311.725462] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14311.750556] LNet: Added LNI 192.168.203.155@tcp50 [8/256/0/180] [14330.337812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14330.364667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14330.384148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14334.437723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14334.450272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14334.455171] Lustre: Skipped 1 previous similar message [14334.487476] Lustre: Skipped 1 previous similar message [14336.324486] Lustre: server umount lustre-MDT0000 complete [14340.488627] LustreError: 473090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787454289 with bad export cookie 15360165951256218306 [14340.493469] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14340.496040] LustreError: 473090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14340.883950] Lustre: server umount lustre-MDT0001 complete [14358.449631] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14360.031128] Lustre: 472371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787454293/real 1787454293] req@00000000b97ce155 x1874281260821888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787454309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14360.067641] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14361.119876] Lustre: 472371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787454294/real 1787454294] req@000000003726a6e4 x1874281260822016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787454310 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14365.151149] Lustre: 472369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787454298/real 1787454298] req@0000000026ce3990 x1874281260822144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787454314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14366.700605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14371.295141] Lustre: 472371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787454304/real 1787454304] req@00000000526e9b8e x1874281260822528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787454320 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14371.324988] Lustre: 472371:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14385.940282] LustreError: 475062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14385.958289] LustreError: 475062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14385.988712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14389.024207] LustreError: 475063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14389.027172] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14389.500678] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14390.055213] LustreError: 475062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14395.362437] LustreError: 475063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14396.866752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14397.133901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14401.498326] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14405.616533] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14409.992657] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14415.911413] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14437.344379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14437.361076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14437.391834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14437.406769] Lustre: Skipped 1 previous similar message [14438.386229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14441.443109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14441.458931] Lustre: Skipped 2 previous similar messages [14443.439241] Lustre: server umount lustre-OST0000 complete [14448.608530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14448.623787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14448.639735] Lustre: Skipped 1 previous similar message [14448.644859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14453.925809] Lustre: server umount lustre-MDT0000 complete [14456.801530] LustreError: 499323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14456.827900] LustreError: 499323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14457.561526] LustreError: 473894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787454406 with bad export cookie 15360165951256220959 [14457.563473] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14457.583648] LustreError: 473894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14458.080103] Lustre: server umount lustre-MDT0001 complete [14469.496476] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14473.523062] Key type lgssc unregistered [14473.902342] LNet: 502124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14473.909085] LNetError: 502124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14473.914532] LNet: Removed LNI 192.168.203.155@tcp [14473.919854] LNet: Removed LNI 192.168.203.155@tcp10 [14473.933655] LNet: Removed LNI 192.168.203.155@tcp11 [14473.935824] LNet: Removed LNI 192.168.203.155@tcp12 [14473.944245] LNet: Removed LNI 192.168.203.155@tcp13 [14473.947184] LNet: Removed LNI 192.168.203.155@tcp14 [14473.953329] LNet: Removed LNI 192.168.203.155@tcp15 [14473.957377] LNet: Removed LNI 192.168.203.155@tcp16 [14473.961587] LNet: Removed LNI 192.168.203.155@tcp17 [14473.964653] LNet: Removed LNI 192.168.203.155@tcp18 [14473.978194] LNet: Removed LNI 192.168.203.155@tcp19 [14473.984214] LNet: Removed LNI 192.168.203.155@tcp20 [14473.989199] LNet: Removed LNI 192.168.203.155@tcp21 [14473.993575] LNet: Removed LNI 192.168.203.155@tcp22 [14474.002795] LNet: Removed LNI 192.168.203.155@tcp23 [14474.011520] LNet: Removed LNI 192.168.203.155@tcp24 [14474.017718] LNet: Removed LNI 192.168.203.155@tcp25 [14474.023909] LNet: Removed LNI 192.168.203.155@tcp26 [14474.029360] LNet: Removed LNI 192.168.203.155@tcp27 [14474.033585] LNet: Removed LNI 192.168.203.155@tcp28 [14474.036442] LNet: Removed LNI 192.168.203.155@tcp29 [14474.039411] LNet: Removed LNI 192.168.203.155@tcp30 [14474.042574] LNet: Removed LNI 192.168.203.155@tcp31 [14474.045611] LNet: Removed LNI 192.168.203.155@tcp32 [14474.048383] LNet: Removed LNI 192.168.203.155@tcp33 [14474.051353] LNet: Removed LNI 192.168.203.155@tcp34 [14474.055666] LNet: Removed LNI 192.168.203.155@tcp35 [14474.059178] LNet: Removed LNI 192.168.203.155@tcp36 [14474.062901] LNet: Removed LNI 192.168.203.155@tcp37 [14474.064925] LNet: Removed LNI 192.168.203.155@tcp38 [14474.067487] LNet: Removed LNI 192.168.203.155@tcp39 [14474.069755] LNet: Removed LNI 192.168.203.155@tcp40 [14474.072325] LNet: Removed LNI 192.168.203.155@tcp41 [14474.074944] LNet: Removed LNI 192.168.203.155@tcp42 [14474.077378] LNet: Removed LNI 192.168.203.155@tcp43 [14474.079941] LNet: Removed LNI 192.168.203.155@tcp44 [14474.082262] LNet: Removed LNI 192.168.203.155@tcp45 [14474.084880] LNet: Removed LNI 192.168.203.155@tcp46 [14474.087395] LNet: Removed LNI 192.168.203.155@tcp47 [14474.089726] LNet: Removed LNI 192.168.203.155@tcp48 [14474.092248] LNet: Removed LNI 192.168.203.155@tcp49 [14474.103625] LNet: Removed LNI 192.168.203.155@tcp50 [14474.840767] Key type .llcrypt unregistered [14474.846184] Key type ._llcrypt unregistered [14496.848071] Key type ._llcrypt registered [14496.851542] Key type .llcrypt registered [14497.004392] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [14511.459824] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14512.626289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14512.660720] alg: No test for adler32 (adler32-zlib) [14513.734628] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [14514.005712] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14515.719168] Key type lgssc registered [14516.877517] Lustre: Echo OBD driver; http://www.lustre.org/ [14548.465384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14548.495465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14549.819350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14549.862334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14549.966781] Lustre: lustre-MDT0000: new disk, initializing [14550.054517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14550.073636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14554.565475] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14564.865622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14565.010618] Lustre: 505320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14565.090737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14565.095518] Lustre: Skipped 1 previous similar message [14565.155038] Lustre: lustre-MDT0001: new disk, initializing [14565.245943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14565.285622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14565.300791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14569.960634] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14579.868943] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14586.058140] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14593.310743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14593.513072] Lustre: *** cfs_fail_loc=90f, val=0*** [14593.611367] Lustre: lustre-OST0000: new disk, initializing [14593.616435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14593.624951] Lustre: 507337:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14593.714534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14600.046366] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14601.375667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14601.392090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14601.551362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14610.344809] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14622.184073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14622.190798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14622.205669] LustreError: Skipped 1 previous similar message [14622.223906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14624.213566] Lustre: server umount lustre-OST0000 complete [14631.905515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14631.917611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14631.926189] Lustre: Skipped 1 previous similar message [14631.930193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14631.932928] Lustre: Skipped 1 previous similar message [14634.059748] Lustre: server umount lustre-MDT0000 complete [14637.774744] LustreError: 505313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787454586 with bad export cookie 3343925202487117999 [14637.782916] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14637.784705] LustreError: 505313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14638.172115] Lustre: server umount lustre-MDT0001 complete [14649.573228] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14652.140894] Key type lgssc unregistered [14652.456208] LNet: 509247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14652.464195] LNetError: 509247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14652.493082] LNet: Removed LNI 192.168.203.155@tcp [14653.322758] Key type .llcrypt unregistered [14653.325028] Key type ._llcrypt unregistered [14671.942579] Key type ._llcrypt registered [14671.945086] Key type .llcrypt registered [14672.058450] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14673.055376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14673.149246] alg: No test for adler32 (adler32-zlib) [14674.244616] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [14674.636731] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14676.343125] Key type lgssc registered [14677.516650] Lustre: Echo OBD driver; http://www.lustre.org/ [14684.928069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14684.957423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14686.350901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14690.452344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14696.930245] LustreError: 510519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14696.951893] LustreError: 510519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14698.911732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14699.391068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14704.078780] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14712.340615] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14718.842704] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14726.109194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14726.477762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14732.892326] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14740.604353] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14755.271775] loop0: detected capacity change from 0 to 204800000 [14755.541174] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14755.551763] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14755.589505] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14762.569052] loop0: detected capacity change from 0 to 204800000 [14762.866142] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14762.873464] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14762.889842] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14770.880400] loop0: detected capacity change from 0 to 204800000 [14777.721736] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14777.882173] Lustre: fs15246-OST0000: new disk, initializing [14777.893621] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14777.903323] Lustre: 514113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14777.962853] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14785.548845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14795.985242] loop1: detected capacity change from 0 to 204800000 [14803.793227] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14803.884117] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14803.979728] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14804.109224] Lustre: fs15246-MDT0000: new disk, initializing [14804.280583] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14804.311063] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14804.457796] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14805.536696] Lustre: 515237:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000062bbc7cb x1874282018817920/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787454770 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14809.220824] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14814.627651] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14814.643536] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14814.668772] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14834.151756] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14834.172127] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14836.669633] Lustre: server umount fs15246-MDT0000 complete [14839.273618] LustreError: 510515:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14844.390400] LustreError: 510513:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14846.971905] Lustre: server umount fs15246-OST0000 complete [14862.820444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14862.837204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14862.871780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14864.880327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14864.882889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14864.921942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14867.449193] Lustre: server umount lustre-OST0000 complete [14873.585963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14873.607214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14873.657936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14873.662959] Lustre: Skipped 1 previous similar message [14877.991654] Lustre: server umount lustre-MDT0000 complete [14880.228193] LustreError: 510518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14880.250486] LustreError: 510518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14881.592487] LustreError: 510499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787454830 with bad export cookie 5995740791961363779 [14881.600151] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14881.609708] LustreError: 510499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14881.932846] Lustre: server umount lustre-MDT0001 complete [14892.958874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14895.606388] Key type lgssc unregistered [14895.905738] LNet: 518338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14895.924355] LNetError: 518338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14895.944111] LNet: Removed LNI 192.168.203.155@tcp [14896.791640] Key type .llcrypt unregistered [14896.793948] Key type ._llcrypt unregistered [14916.216378] Key type ._llcrypt registered [14916.219664] Key type .llcrypt registered [14916.361738] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14917.294209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14917.448076] alg: No test for adler32 (adler32-zlib) [14918.517566] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [14918.747294] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14920.391724] Key type lgssc registered [14921.503932] Lustre: Echo OBD driver; http://www.lustre.org/ [14955.923606] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [14965.677054] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14998.508649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14998.574416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14999.758753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14999.787561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14999.857917] Lustre: lustre-MDT0000: new disk, initializing [14999.924991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14999.946979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15004.250955] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15013.876304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15014.035543] Lustre: 522804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15014.081795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15014.088267] Lustre: Skipped 1 previous similar message [15014.197911] Lustre: lustre-MDT0001: new disk, initializing [15014.324449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15014.374702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15014.385723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15018.392232] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15027.564415] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15033.736714] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15040.169555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15040.458751] Lustre: lustre-OST0000: new disk, initializing [15040.470706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15040.482104] Lustre: 524773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15040.608311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15047.156792] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15050.338225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15050.360861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15050.457644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15056.682436] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15061.439622] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15061.617229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15065.747965] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15065.947875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15071.208226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15071.214583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15071.215216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15071.228094] Lustre: Skipped 1 previous similar message [15071.249840] LustreError: Skipped 1 previous similar message [15073.892827] Lustre: server umount lustre-OST0000 complete [15080.929764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15080.942338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15080.960521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15080.968261] Lustre: Skipped 1 previous similar message [15083.882167] Lustre: server umount lustre-MDT0000 complete [15087.857684] LustreError: 522796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787455036 with bad export cookie 12227765205863493308 [15087.871309] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15088.443760] Lustre: server umount lustre-MDT0001 complete [15117.114982] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [15128.658534] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [15157.621417] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14835 sec ========= 23:18:25 (1787455105) [15159.692239] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:18:26 (1787455106) === [15163.324561] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:18:30 (1787455110) === [15202.284331] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [15205.064768] Key type lgssc unregistered [15205.297211] LNet: 531420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15205.305395] LNetError: 531420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15205.320751] LNet: Removed LNI 192.168.203.155@tcp [15206.346222] Key type .llcrypt unregistered [15206.354465] Key type ._llcrypt unregistered