[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477080490 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2480MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002328] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007359] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013087] *** VALIDATE selinux *** [ 0.021611] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026251] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028117] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.032036] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035312] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037088] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.041005] Speculative Store Bypass: Vulnerable [ 0.044325] debug: unmapping init [mem 0xffffffff84659000-0xffffffff84660fff] [ 0.046967] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047689] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052011] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054010] ... event mask: 000000070000000f [ 0.055329] rcu: Hierarchical SRCU implementation. [ 0.057532] smp: Bringing up secondary CPUs ... [ 0.058648] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.062586] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141297] node 0 deferred pages initialised in 74ms [ 0.143121] devtmpfs: initialized [ 0.144171] x86/mm: Memory block size: 128MB [ 0.146202] gcov: version magic: 0x41383552 [ 0.148231] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152273] pinctrl core: initialized pinctrl subsystem [ 0.153139] [ 0.153476] ************************************************************* [ 0.155009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156011] ** ** [ 0.158008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159008] ** ** [ 0.161009] ** This means that this kernel is built to expose internal ** [ 0.162014] ** IOMMU data structures, which may compromise security on ** [ 0.164013] ** your system. ** [ 0.166009] ** ** [ 0.167011] ** If you see this message and you are not debugging the ** [ 0.169058] ** kernel, report this immediately to your vendor! ** [ 0.170011] ** ** [ 0.171010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173058] ************************************************************* [ 0.175739] NET: Registered protocol family 16 [ 0.176510] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.183046] cpuidle: using governor menu [ 0.184930] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.187693] PCI: Using configuration type 1 for base access [ 0.189227] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198138] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.200033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202171] cryptd: max_cpu_qlen set to 1000 [ 0.204154] ACPI: Added _OSI(Module Device) [ 0.205010] ACPI: Added _OSI(Processor Device) [ 0.207012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214040] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220472] ACPI: Interpreter enabled [ 0.221059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223012] ACPI: Using IOAPIC for interrupt routing [ 0.224117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237493] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.239036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.247293] acpiphp: Slot [2] registered [ 0.248132] acpiphp: Slot [5] registered [ 0.249045] acpiphp: Slot [6] registered [ 0.249954] acpiphp: Slot [7] registered [ 0.251094] acpiphp: Slot [8] registered [ 0.252097] acpiphp: Slot [9] registered [ 0.253136] acpiphp: Slot [10] registered [ 0.255097] acpiphp: Slot [3] registered [ 0.256091] acpiphp: Slot [4] registered [ 0.257107] acpiphp: Slot [11] registered [ 0.258134] acpiphp: Slot [12] registered [ 0.260073] acpiphp: Slot [13] registered [ 0.260808] acpiphp: Slot [14] registered [ 0.261051] acpiphp: Slot [15] registered [ 0.262087] acpiphp: Slot [16] registered [ 0.262873] acpiphp: Slot [17] registered [ 0.264059] acpiphp: Slot [18] registered [ 0.264816] acpiphp: Slot [19] registered [ 0.265052] acpiphp: Slot [20] registered [ 0.265798] acpiphp: Slot [21] registered [ 0.267074] acpiphp: Slot [22] registered [ 0.267837] acpiphp: Slot [23] registered [ 0.268046] acpiphp: Slot [24] registered [ 0.268822] acpiphp: Slot [25] registered [ 0.270055] acpiphp: Slot [26] registered [ 0.270770] acpiphp: Slot [27] registered [ 0.271052] acpiphp: Slot [28] registered [ 0.271821] acpiphp: Slot [29] registered [ 0.273074] acpiphp: Slot [30] registered [ 0.274080] acpiphp: Slot [31] registered [ 0.275046] PCI host bridge to bus 0000:00 [ 0.275901] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.278018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.280015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.281021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.283039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.285166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.288168] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.290562] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.298015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.302651] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.305018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.306012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.308019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.310358] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.311439] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.313033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.315513] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.317961] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.326989] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.331016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.335179] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.341019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.345017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.361022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.371728] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.380018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.386015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.404019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.422175] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.427017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.433019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.445020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.455120] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.459013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.463015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.478017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.485000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.491018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.497019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.513022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.523582] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.529016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.533017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.544023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.552970] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.555468] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.558638] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.560474] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.562202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.567047] iommu: Default domain type: Passthrough [ 0.569634] SCSI subsystem initialized [ 0.571213] ACPI: bus type USB registered [ 0.572109] usbcore: registered new interface driver usbfs [ 0.574109] usbcore: registered new interface driver hub [ 0.576106] usbcore: registered new device driver usb [ 0.578164] pps_core: LinuxPPS API ver. 1 registered [ 0.579011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.582059] PTP clock support registered [ 0.584088] EDAC MC: Ver: 3.0.0 [ 0.586151] PCI: Using ACPI for IRQ routing [ 0.587888] NetLabel: Initializing [ 0.589015] NetLabel: domain hash size = 128 [ 0.590009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.591064] NetLabel: unlabeled traffic allowed by default [ 0.593130] vgaarb: loaded [ 0.594206] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.596012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.603255] clocksource: Switched to clocksource kvm-clock [ 0.710367] VFS: Disk quotas dquot_6.6.0 [ 0.711790] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.714151] *** VALIDATE ramfs *** [ 0.715319] *** VALIDATE hugetlbfs *** [ 0.716578] pnp: PnP ACPI init [ 0.718375] pnp: PnP ACPI: found 6 devices [ 0.736059] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.739087] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.740685] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.742568] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.744551] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.746584] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.748802] NET: Registered protocol family 2 [ 0.751071] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.755472] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.758557] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.763549] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.766695] TCP: Hash tables configured (established 65536 bind 65536) [ 0.769208] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.771956] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.774530] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.777197] NET: Registered protocol family 1 [ 0.780238] RPC: Registered named UNIX socket transport module. [ 0.782154] RPC: Registered udp transport module. [ 0.783667] RPC: Registered tcp transport module. [ 0.785263] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.786834] NET: Registered protocol family 44 [ 0.787976] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.790045] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.791897] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.793380] PCI: CLS 0 bytes, default 64 [ 0.794377] Unpacking initramfs... [ 2.147894] debug: unmapping init [mem 0xffff9b777cc54000-0xffff9b777ffbffff] [ 2.151257] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.152816] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.155541] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.649313] Initialise system trusted keyrings [ 2.650906] Key type blacklist registered [ 2.652943] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.661695] zbud: loaded [ 2.664890] *** VALIDATE nfs *** [ 2.666109] *** VALIDATE nfs4 *** [ 2.667770] pstore: using deflate compression [ 2.671063] Platform Keyring initialized [ 2.774736] NET: Registered protocol family 38 [ 2.776410] Key type asymmetric registered [ 2.777902] Asymmetric key parser 'x509' registered [ 2.779577] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.782250] io scheduler mq-deadline registered [ 2.783632] io scheduler kyber registered [ 2.785215] io scheduler bfq registered [ 2.786685] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.789404] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.791839] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.794561] ACPI: Power Button [PWRF] [ 2.799743] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.806782] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.825291] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.833598] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.855790] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.885563] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.915124] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.920687] Non-volatile memory driver v1.3 [ 2.922150] Linux agpgart interface v0.103 [ 2.953483] virtio_blk virtio1: [vda] 136536 512-byte logical blocks (69.9 MB/66.7 MiB) [ 2.956219] vda: detected capacity change from 0 to 69906432 [ 2.975665] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.978365] vdb: detected capacity change from 0 to 1073741824 [ 2.997249] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.999583] vdc: detected capacity change from 0 to 2621440000 [ 3.013604] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.016429] vdd: detected capacity change from 0 to 2621440000 [ 3.035803] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.038319] vde: detected capacity change from 0 to 4294967296 [ 3.056591] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.059480] vdf: detected capacity change from 0 to 4294967296 [ 3.065606] libphy: Fixed MDIO Bus: probed [ 3.073220] usbcore: registered new interface driver usbserial_generic [ 3.075652] usbserial: USB Serial support registered for generic [ 3.077833] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.081696] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.083390] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.085640] mousedev: PS/2 mouse device common for all mice [ 3.088919] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.089222] rtc_cmos 00:05: RTC can wake from S4 [ 3.096343] rtc_cmos 00:05: registered as rtc0 [ 3.097383] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.098807] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.098842] intel_pstate: CPU model not supported [ 3.102983] hid: raw HID events driver (C) Jiri Kosina [ 3.110195] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.112377] usbcore: registered new interface driver usbhid [ 3.112385] usbhid: USB HID core driver [ 3.112508] drop_monitor: Initializing network drop monitor service [ 3.112636] Initializing XFRM netlink socket [ 3.113056] NET: Registered protocol family 10 [ 3.114276] Segment Routing with IPv6 [ 3.128691] NET: Registered protocol family 17 [ 3.130813] mpls_gso: MPLS GSO support [ 3.136656] RAS: Correctable Errors collector initialized. [ 3.138719] AVX version of gcm_enc/dec engaged. [ 3.140299] AES CTR mode by8 optimization enabled [ 3.211519] sched_clock: Marking stable (3211487692, 0)->(4051334670, -839846978) [ 3.215217] registered taskstats version 1 [ 3.217166] Loading compiled-in X.509 certificates [ 3.218832] zswap: loaded using pool lzo/zbud [ 3.244127] Key type big_key registered [ 3.257354] Key type encrypted registered [ 3.258883] ima: No TPM chip found, activating TPM-bypass! [ 3.260855] ima: Allocated hash algorithm: sha1 [ 3.262318] ima: No architecture policies found [ 3.263813] evm: Initialising EVM extended attributes: [ 3.265611] evm: security.selinux [ 3.266646] evm: security.ima [ 3.267632] evm: security.capability [ 3.268676] evm: HMAC attrs: 0x1 [ 3.270669] rtc_cmos 00:05: setting system clock to 2026-06-12 00:56:58 UTC (1781225818) [ 3.276234] debug: unmapping init [mem 0xffffffff85603000-0xffffffff857fffff] [ 3.279057] debug: unmapping init [mem 0xffffffff84382000-0xffffffff84658fff] [ 3.287114] Write protecting the kernel read-only data: 28672k [ 3.290320] debug: unmapping init [mem 0xffffffff82a03000-0xffffffff82bfffff] [ 3.292214] debug: unmapping init [mem 0xffffffff83314000-0xffffffff833fffff] [ 3.325388] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.333356] systemd[1]: Detected virtualization kvm. [ 3.335200] systemd[1]: Detected architecture x86-64. [ 3.336994] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.364380] systemd[1]: No hostname configured. [ 3.365659] systemd[1]: Set hostname to . [ 3.367041] random: systemd: uninitialized urandom read (16 bytes read) [ 3.369380] systemd[1]: Initializing machine ID from random generator. [ 3.407886] random: ln: uninitialized urandom read (6 bytes read) [ 3.489657] random: systemd: uninitialized urandom read (16 bytes read) [ 3.492168] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.495191] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.497654] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.996186] device-mapper: uevent: version 1.0.3 [ 3.997610] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.623889] random: fast init done [ 4.626984] virtio_net virtio0 ens2: renamed from eth0 [ 4.707959] scsi host0: ata_piix [ 4.732674] scsi host1: ata_piix [ 4.733657] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.735519] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.246472] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.773477] random: crng init done [ 9.774766] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.006601] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.073578] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.327355] SELinux: Disabled at runtime. [ 11.384621] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.391063] systemd[1]: Detected virtualization kvm. [ 11.392627] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.833191] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.836304] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.840708] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.844616] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.848501] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.855478] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.861467] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ 11.905968] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.195132] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.499345] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.550095] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.670117] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.682693] EDAC sbridge: Ver: 1.1.2 [ 13.759827] Key type dns_resolver registered [ 14.037204] NFS: Registering the id_resolver key type [ 14.038625] Key type id_resolver registered [ 14.039563] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg115-server login: [ 40.245426] libcfs: loading out-of-tree module taints kernel. [ 40.265338] Key type ._llcrypt registered [ 40.266793] Key type .llcrypt registered [ 40.306486] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 48.690888] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 49.261942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.268281] alg: No test for adler32 (adler32-zlib) [ 50.318779] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 50.735400] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 52.375202] Key type lgssc registered [ 53.071202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 61.701365] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 63.859008] hrtimer: interrupt took 5412160 ns [ 107.840517] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 122.445814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 122.502753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 123.898603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 123.924910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 124.007992] Lustre: lustre-MDT0000: new disk, initializing [ 124.069230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 124.085300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 129.605515] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 145.081879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.259671] Lustre: 6491:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 145.308331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 145.312116] Lustre: Skipped 1 previous similar message [ 145.424624] Lustre: lustre-MDT0001: new disk, initializing [ 145.539557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 145.607104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 145.625442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 150.560870] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.992343] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 166.777408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 167.230959] Lustre: lustre-OST0000: new disk, initializing [ 167.237680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 167.249807] Lustre: 8400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 167.333104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.570524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 172.588164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 172.636226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 174.789385] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.360784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.514348] Lustre: lustre-OST0001: new disk, initializing [ 192.520527] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 192.538396] Lustre: 9454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.637116] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.577262] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.791796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 201.816692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 201.974059] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 212.748760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.411589] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 230.171975] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing check_logdir /tmp/testlogs/ [ 235.090274] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing yml_node [ 239.535579] Lustre: DEBUG MARKER: Client: 2.17.53.68 [ 241.892766] Lustre: DEBUG MARKER: MDS: 2.17.53.68 [ 244.595819] Lustre: DEBUG MARKER: OSS: 2.17.53.68 [ 246.830543] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 21:01:00 EDT 2026 [ 266.610109] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 267.976499] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 289.258446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 289.259608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.264924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.307106] Lustre: Skipped 3 previous similar messages [ 290.951674] Lustre: server umount lustre-MDT0000 complete [ 294.368792] LustreError: 8399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.398039] LustreError: 8399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 294.506690] LustreError: 9453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226109 with bad export cookie 11269712253322952142 [ 294.514343] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.516718] LustreError: 9453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 299.487897] LustreError: 6499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.491246] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.492019] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.492024] Lustre: Skipped 3 previous similar messages [ 299.501761] LustreError: 6499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 299.508412] Lustre: Skipped 1 previous similar message [ 301.067578] Lustre: server umount lustre-MDT0001 complete [ 305.240373] Lustre: server umount lustre-OST0000 complete [ 308.762725] Lustre: server umount lustre-OST0001 complete [ 314.981810] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 324.710539] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 360.687245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.040126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 361.082307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 361.173369] Lustre: lustre-MDT0000: new disk, initializing [ 361.298363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 361.318972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 366.119979] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 378.669056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.752879] Lustre: 14991:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 378.835084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 378.839703] Lustre: Skipped 1 previous similar message [ 378.879668] Lustre: lustre-MDT0001: new disk, initializing [ 379.029292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 379.096175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 379.102981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 383.885748] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 392.524573] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 397.568757] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 405.411946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 405.754390] Lustre: lustre-OST0000: new disk, initializing [ 405.764654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 405.784265] Lustre: 16734:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 405.925113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.679175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 407.696305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 407.864573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 412.325603] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 421.363758] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 426.484483] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 426.739503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 431.286927] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 431.467040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 433.632519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 433.635489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.644769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 433.644778] Lustre: Skipped 1 previous similar message [ 433.665447] Lustre: Skipped 1 previous similar message [ 438.754928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 438.760441] Lustre: Skipped 1 previous similar message [ 439.598312] Lustre: server umount lustre-OST0000 complete [ 443.885829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.899741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 443.902380] Lustre: Skipped 1 previous similar message [ 450.141243] Lustre: server umount lustre-MDT0000 complete [ 453.981267] LustreError: 14982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226269 with bad export cookie 11269712253322953675 [ 453.982690] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.014609] LustreError: 14982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 454.520812] Lustre: server umount lustre-MDT0001 complete [ 458.733794] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:04:32 (1781226272) [ 461.167655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 462.776719] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:04:36 (1781226276) [ 469.553687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.148386] LustreError: 18996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.204446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.541619] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 475.625531] LustreError: 18997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.739431] LustreError: 18996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.639687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.378346] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 493.249461] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 496.926947] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 503.606148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 510.356354] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 515.655388] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 523.756308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.769069] LustreError: Skipped 1 previous similar message [ 523.779052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.792976] Lustre: Skipped 1 previous similar message [ 523.815387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.826703] Lustre: Skipped 2 previous similar messages [ 525.793365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.913462] Lustre: server umount lustre-OST0000 complete [ 533.990064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.003065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.037442] Lustre: Skipped 1 previous similar message [ 539.677426] Lustre: server umount lustre-MDT0000 complete [ 541.157279] LustreError: 18997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.187699] LustreError: 18997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 544.125775] LustreError: 18977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226359 with bad export cookie 11269712253322954557 [ 544.131581] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.132033] LustreError: 18977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 544.795775] Lustre: server umount lustre-MDT0001 complete [ 556.463106] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 559.515764] Key type lgssc unregistered [ 559.905744] LNet: 22170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.911842] LNetError: 22170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.929795] LNet: Removed LNI 192.168.201.115@tcp [ 561.153231] Key type .llcrypt unregistered [ 561.154563] Key type ._llcrypt unregistered [ 587.584314] Key type ._llcrypt registered [ 587.586814] Key type .llcrypt registered [ 587.678554] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 589.344150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 589.407800] alg: No test for adler32 (adler32-zlib) [ 590.510521] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 590.802501] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 592.471928] Key type lgssc registered [ 593.912135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.871425] loop: module loaded [ 601.949704] loop0: detected capacity change from 0 to 204800000 [ 602.137192] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.142762] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.374212] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 607.534566] loop0: detected capacity change from 0 to 204800000 [ 607.685501] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 607.695505] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 607.714770] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 613.202666] loop0: detected capacity change from 0 to 204800000 [ 619.652375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 619.701968] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.974461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 621.047193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 621.102725] Lustre: lustre-MDT0000: new disk, initializing [ 621.179953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.203441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 626.177806] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 635.275658] loop1: detected capacity change from 0 to 204800000 [ 642.632071] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 643.053321] Lustre: lustre-OST07c6: new disk, initializing [ 643.058901] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 643.064939] Lustre: Skipped 1 previous similar message [ 643.069238] Lustre: 25322:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 643.164745] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 649.565364] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 651.923857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 651.939590] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 651.993794] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 655.142674] Lustre: 24327:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 655.152810] Lustre: 24327:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 655.160744] Lustre: 24327:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 655.171823] Lustre: 24327:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 655.186432] Lustre: 24327:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 655.197415] Lustre: 24327:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 673.173113] Lustre: 24327:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 673.185730] Lustre: 24327:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 673.193437] Lustre: 24327:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 673.203535] Lustre: 24327:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 673.214423] Lustre: 24327:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 673.220864] Lustre: 24327:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 678.145993] Lustre: 25889:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 678.156846] Lustre: 25889:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 678.164791] Lustre: 25889:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 678.171192] Lustre: 25889:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.176838] Lustre: 25889:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 678.184494] Lustre: 25889:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.193460] Lustre: 25889:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 678.198709] Lustre: 25889:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.201558] Lustre: 25889:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 678.207368] Lustre: 25889:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.215417] Lustre: 25889:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 678.222253] Lustre: 25889:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.518899] Lustre: 24326:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 681.537761] Lustre: 24326:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 681.548838] Lustre: 24326:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 681.558633] Lustre: 24326:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.573657] Lustre: 24326:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 681.597431] Lustre: 24326:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.630070] Lustre: 24326:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 681.641401] Lustre: 24326:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.658750] Lustre: 24326:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 681.673330] Lustre: 24326:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.687308] Lustre: 24326:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 681.696620] Lustre: 24326:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.870195] Lustre: 24326:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 686.880387] Lustre: 24326:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 686.887337] Lustre: 24326:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 686.894662] Lustre: 24326:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.908290] Lustre: 24326:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.928033] Lustre: 24326:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.934088] Lustre: 24326:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 686.941487] Lustre: 24326:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.950520] Lustre: 24326:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.959789] Lustre: 24326:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.973317] Lustre: 24326:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.988608] Lustre: 24326:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 695.851432] Lustre: 25889:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 695.878209] Lustre: 25889:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 695.880650] Lustre: 25889:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 695.900939] Lustre: 25889:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 695.906924] Lustre: 25889:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 695.915605] Lustre: 25889:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 695.925808] Lustre: 25889:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 695.943802] Lustre: 25889:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 695.961029] Lustre: 25889:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 695.978640] Lustre: 25889:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 695.989631] Lustre: 25889:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.010428] Lustre: 25889:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 711.932296] Lustre: 24327:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 711.945501] Lustre: 24327:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 711.949988] Lustre: 24327:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 711.954126] Lustre: 24327:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 711.958531] Lustre: 24327:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 711.962648] Lustre: 24327:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 711.968171] Lustre: 24327:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 711.973731] Lustre: 24327:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 711.977858] Lustre: 24327:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 711.984604] Lustre: 24327:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 711.991762] Lustre: 24327:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 711.998261] Lustre: 24327:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 749.033110] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.054255] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 754.146792] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 759.264847] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 762.335153] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 762.708748] Lustre: server umount lustre-OST07c6 complete [ 773.259944] Lustre: server umount lustre-MDT0000 complete [ 790.000720] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 792.965793] Key type lgssc unregistered [ 793.314988] LNet: 27847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.324401] LNetError: 27847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.349473] LNet: Removed LNI 192.168.201.115@tcp [ 794.192324] Key type .llcrypt unregistered [ 794.194827] Key type ._llcrypt unregistered [ 814.369319] Key type ._llcrypt registered [ 814.378816] Key type .llcrypt registered [ 814.563159] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 815.742649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 815.946273] alg: No test for adler32 (adler32-zlib) [ 817.243733] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 817.627497] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 819.424751] Key type lgssc registered [ 820.483835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 832.576394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 832.615841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.353237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 839.187529] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 844.773641] LustreError: 29104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.795784] LustreError: 29104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 849.294201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.574722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 854.106939] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 859.824876] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 864.046550] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 870.582056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 870.898733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.495360] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 879.127477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 882.698579] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 904.673092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 904.682503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.684594] LustreError: Skipped 1 previous similar message [ 904.686502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 904.734298] Lustre: Skipped 1 previous similar message [ 908.623874] Lustre: server umount lustre-OST0000 complete [ 914.914889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.926892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.929316] Lustre: Skipped 1 previous similar message [ 918.848718] Lustre: server umount lustre-MDT0000 complete [ 920.037096] LustreError: 30876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.062750] LustreError: 30876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 924.049576] LustreError: 29084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226739 with bad export cookie 1025824798982458136 [ 924.070087] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 925.138318] Lustre: server umount lustre-MDT0001 complete [ 937.532910] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 940.244471] Key type lgssc unregistered [ 940.626815] LNet: 32484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.647659] LNetError: 32484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.674613] LNet: Removed LNI 192.168.201.115@tcp [ 941.802122] Key type .llcrypt unregistered [ 941.803642] Key type ._llcrypt unregistered [ 964.217765] Key type ._llcrypt registered [ 964.220532] Key type .llcrypt registered [ 964.327461] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 965.871816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 965.909308] alg: No test for adler32 (adler32-zlib) [ 966.923715] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 967.115609] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 968.768096] Key type lgssc registered [ 970.162125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 979.974154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 980.007686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.885730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.989842] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 992.237976] LustreError: 33740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 992.261985] LustreError: 33740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 993.253401] LustreError: 33739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.376797] LustreError: 33740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.843333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.560869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1006.626162] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1013.757990] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1018.310311] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1025.320805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1025.718548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.798173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1032.112344] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1038.274293] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1054.691285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1054.700627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.726121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1055.713591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1055.728805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1055.738112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.879428] Lustre: server umount lustre-OST0000 complete [ 1065.954367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.959970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1065.987611] Lustre: Skipped 1 previous similar message [ 1066.019213] Lustre: Skipped 2 previous similar messages [ 1071.079160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.193292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.202366] Lustre: Skipped 2 previous similar messages [ 1078.240980] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1080.491452] Lustre: server umount lustre-MDT0000 complete [ 1081.312946] LustreError: 33739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.312946] LustreError: 33734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.740932] LustreError: 33720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226898 with bad export cookie 10028038820130402493 [ 1083.755102] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1083.759418] LustreError: 33720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1084.371579] Lustre: server umount lustre-MDT0001 complete [ 1096.485676] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1099.902873] Key type lgssc unregistered [ 1100.140267] LNet: 36939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.159870] LNetError: 36939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.200337] LNet: Removed LNI 192.168.201.115@tcp [ 1101.225478] Key type .llcrypt unregistered [ 1101.228571] Key type ._llcrypt unregistered [ 1127.668780] Key type ._llcrypt registered [ 1127.671393] Key type .llcrypt registered [ 1127.785328] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1129.561891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1129.607315] alg: No test for adler32 (adler32-zlib) [ 1131.517726] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 1131.895617] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1133.775252] Key type lgssc registered [ 1135.635822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1144.461652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1144.496454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.899028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.878855] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1156.077598] LustreError: 38337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.113944] LustreError: 38337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1161.185179] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.538352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.052131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1167.847986] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1175.275612] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1179.703601] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1187.105802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1187.554050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.735645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1195.260890] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1202.475660] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1209.430831] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 1212.388109] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.391444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1212.419122] Lustre: Skipped 1 previous similar message [ 1212.442480] Lustre: Skipped 2 previous similar messages [ 1214.519635] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 1214.965686] Lustre: server umount lustre-MDT0000 complete [ 1217.506129] LustreError: 38332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.517209] LustreError: 38332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1219.152763] LustreError: 38319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227034 with bad export cookie 7429559498182522200 [ 1219.165175] LustreError: 38319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1219.176046] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1219.630252] LustreError: 38334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.659677] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [ 1222.645695] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.657571] Lustre: Skipped 1 previous similar message [ 1224.672747] LustreError: 38333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1224.673897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1224.710969] LustreError: 38333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1224.747790] Lustre: Skipped 1 previous similar message [ 1225.879211] Lustre: server umount lustre-MDT0001 complete [ 1247.703610] Lustre: server umount lustre-OST0000 complete [ 1265.143916] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1268.277420] Key type lgssc unregistered [ 1268.579637] LNet: 41844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.589246] LNetError: 41844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.603846] LNet: Removed LNI 192.168.201.115@tcp [ 1269.525930] Key type .llcrypt unregistered [ 1269.527321] Key type ._llcrypt unregistered [ 1296.945840] Key type ._llcrypt registered [ 1296.953638] Key type .llcrypt registered [ 1297.099476] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1298.675950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1298.890484] alg: No test for adler32 (adler32-zlib) [ 1299.925838] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 1300.144838] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1301.943519] Key type lgssc registered [ 1303.572657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1312.508359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1312.550976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.236960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.684471] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1324.512313] LustreError: 43245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.536096] LustreError: 43245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1328.202910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.661482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1334.470237] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1340.725542] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1344.126665] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1351.857025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1352.290453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.495484] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1359.549651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1366.344395] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1373.260180] Lustre: lustre-OST0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 1374.178108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1374.191650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.210186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1375.211578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1375.215263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.218290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1378.349162] Lustre: lustre-OST0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 1378.360221] Lustre: Skipped 1 previous similar message [ 1378.690584] Lustre: server umount lustre-OST0000 complete [ 1400.291303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1400.305542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.337841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1404.306790] Lustre: server umount lustre-MDT0000 complete [ 1406.949951] LustreError: 43245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.964959] LustreError: 43245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1408.028980] LustreError: 43225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227223 with bad export cookie 12496355773280033394 [ 1408.050532] LustreError: 43225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1408.055107] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1414.777832] Lustre: server umount lustre-MDT0001 complete [ 1426.711190] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1429.547617] Key type lgssc unregistered [ 1429.912471] LNet: 46595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.919386] LNetError: 46595:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.933602] LNet: Removed LNI 192.168.201.115@tcp [ 1430.826892] Key type .llcrypt unregistered [ 1430.828863] Key type ._llcrypt unregistered [ 1457.343688] Key type ._llcrypt registered [ 1457.346726] Key type .llcrypt registered [ 1457.468724] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1458.751101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1458.835207] alg: No test for adler32 (adler32-zlib) [ 1459.887913] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 1460.056310] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1461.695231] Key type lgssc registered [ 1462.661914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1471.204708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1471.217850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.775522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.340933] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1482.214831] LustreError: 47995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.241191] LustreError: 47995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1486.718975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.205463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1491.884563] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1499.028866] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1503.488299] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1511.813457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1511.919959] LustreError: 49591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.411708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.507153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1519.002633] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1525.944804] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1532.715047] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1537.413861] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1550.380852] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1121h22m42s [ 1552.580067] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1558.509296] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.525998] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1558.528207] Lustre: Skipped 1 previous similar message [ 1558.543729] Lustre: Skipped 2 previous similar messages [ 1559.584912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1559.597765] Lustre: Skipped 2 previous similar messages [ 1561.514257] Lustre: server umount lustre-MDT0000 complete [ 1565.670495] LustreError: 47991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.681124] LustreError: 47991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1566.153946] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1571.811535] LustreError: 47996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1571.835620] LustreError: 47996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1575.938466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.157937] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1576.478167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.548510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1577.519315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1583.192202] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1587.146509] Lustre: DEBUG MARKER: Wait for df (36117) ... [ 1589.604689] Lustre: DEBUG MARKER: done [ 1596.900434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.911258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.929673] Lustre: Skipped 1 previous similar message [ 1596.947089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1596.955400] Lustre: Skipped 1 previous similar message [ 1597.925832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.826558] Lustre: server umount lustre-OST0000 complete [ 1608.165418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.178747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.183345] Lustre: Skipped 2 previous similar messages [ 1612.432486] Lustre: server umount lustre-MDT0000 complete [ 1615.333074] LustreError: 47992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1615.348737] LustreError: 47992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1616.127736] LustreError: 47977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227431 with bad export cookie 4886339651037736712 [ 1616.133784] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.142967] LustreError: 47977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1616.506339] Lustre: server umount lustre-MDT0001 complete [ 1627.899601] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1630.806566] Key type lgssc unregistered [ 1631.148818] LNet: 52558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.152507] LNetError: 52558:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.168490] LNet: Removed LNI 192.168.201.115@tcp [ 1632.049391] Key type .llcrypt unregistered [ 1632.051578] Key type ._llcrypt unregistered [ 1676.829293] Key type ._llcrypt registered [ 1676.832189] Key type .llcrypt registered [ 1677.027635] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1678.253845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1678.653642] alg: No test for adler32 (adler32-zlib) [ 1679.954926] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 1680.341924] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1682.041226] Key type lgssc registered [ 1682.898430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1691.281751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1691.311290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.657854] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1692.709196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1692.892936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.132840] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1708.489779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.666222] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1708.700791] Lustre: 54758:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1709.002744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1713.647117] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1720.471230] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1725.772944] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1735.065043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1735.514879] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1736.023436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.879561] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1744.064768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1748.886731] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1756.101040] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1121h26m09s [ 1757.715164] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1765.335595] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1767.939337] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1768.012968] LustreError: 54765:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1788.520796] Lustre: lustre-MDT0000: Client a1da4783-40bb-4bca-b136-e3bf0527189b (at 192.168.201.15@tcp) reconnecting [ 1808.087233] LustreError: 54765:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1810.473752] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1814.234560] Lustre: DEBUG MARKER: done [ 1821.155437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.158693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1821.160017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1821.168754] Lustre: Skipped 1 previous similar message [ 1826.274608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1826.280668] Lustre: Skipped 2 previous similar messages [ 1831.399898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1831.409142] Lustre: Skipped 1 previous similar message [ 1833.951165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1834.418937] Lustre: server umount lustre-OST0000 complete [ 1841.632951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.638420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1841.665309] Lustre: Skipped 1 previous similar message [ 1848.800711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.804542] Lustre: Skipped 4 previous similar messages [ 1853.407480] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1853.895903] Lustre: server umount lustre-MDT0000 complete [ 1857.876115] LustreError: 54749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227673 with bad export cookie 10342421364306196616 [ 1857.880580] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1857.884314] LustreError: 54749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1858.055963] Lustre: server umount lustre-MDT0001 complete [ 1869.050032] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1873.000216] Key type lgssc unregistered [ 1873.348849] LNet: 58734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.353596] LNetError: 58734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.369751] LNet: Removed LNI 192.168.201.115@tcp [ 1874.501774] Key type .llcrypt unregistered [ 1874.507229] Key type ._llcrypt unregistered [ 1925.383095] Key type ._llcrypt registered [ 1925.385434] Key type .llcrypt registered [ 1925.573751] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1927.534252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1927.584263] alg: No test for adler32 (adler32-zlib) [ 1928.816642] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 1929.275575] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1931.071582] Key type lgssc registered [ 1932.603815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1937.080668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1937.121367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.546762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1938.579621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1938.883340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.948731] Lustre: Failing over lustre-MDT0000 [ 1940.590635] Lustre: server umount lustre-MDT0000 complete [ 1948.265445] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:29:21 (1781227761) [ 1956.784254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.446105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.506797] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1972.378477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.482110] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1972.527403] Lustre: 61514:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1972.812563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1978.476302] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1984.609515] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1989.097497] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1996.651892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1996.801407] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1997.003372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.461275] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2005.721304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2010.936120] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2018.002421] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2026.465977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.472487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.486924] Lustre: Skipped 2 previous similar messages [ 2026.506957] Lustre: Skipped 2 previous similar messages [ 2031.593706] Lustre: server umount lustre-MDT0000 complete [ 2036.710967] LustreError: 61522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.737535] LustreError: 61522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2036.790237] LustreError: 62235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227852 with bad export cookie 4173935695543979734 [ 2036.799114] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.807574] LustreError: 62235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2037.381777] Lustre: server umount lustre-MDT0001 complete [ 2041.636979] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2047.772534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2058.210257] Lustre: 60440:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227857/real 1781227857] req@00000000a4c3ada2 x1867752662933632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2058.263469] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.404862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.896025] LustreError: 63141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.047611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.228573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2064.116811] Lustre: 60437:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227862/real 1781227862] req@000000008fce533f x1867752662933760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2067.105643] LustreError: 63142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.118237] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2067.122206] LustreError: 63142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2067.948175] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2069.152173] Lustre: 60437:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227867/real 1781227867] req@00000000dcf23313 x1867752662934016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2073.569098] Lustre: 60439:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227872/real 1781227872] req@00000000e7e3d141 x1867752662934144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2073.585551] LustreError: 65215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.620125] LustreError: 65215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2076.977375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.443847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2082.585768] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2082.810256] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2088.952514] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2092.923709] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2103.271996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2103.281523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.285579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.298735] Lustre: Skipped 2 previous similar messages [ 2103.300875] Lustre: Skipped 1 previous similar message [ 2108.397454] LustreError: 63141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.423598] LustreError: 63141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2109.023586] Lustre: server umount lustre-MDT0000 complete [ 2113.141490] LustreError: 64088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227928 with bad export cookie 4173935695543982009 [ 2113.145298] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2113.164122] LustreError: 64088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2113.544749] Lustre: server umount lustre-MDT0001 complete [ 2121.622085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2128.864312] Lustre: 60437:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227928/real 1781227928] req@0000000035f47c02 x1867752662960384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.907194] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.443673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.488894] LustreError: 65665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.634391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.716947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2139.552483] Lustre: 60439:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227938/real 1781227938] req@00000000d814e002 x1867752662960768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2139.586437] Lustre: 60439:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2143.959422] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2153.526138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.188072] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2158.742389] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2164.655880] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2168.946720] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2179.044177] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.046542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2179.060542] Lustre: Skipped 1 previous similar message [ 2179.067507] Lustre: Skipped 3 previous similar messages [ 2184.164931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2184.177445] Lustre: Skipped 2 previous similar messages [ 2189.284244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.291758] Lustre: Skipped 2 previous similar messages [ 2190.303137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2190.525146] Lustre: server umount lustre-MDT0000 complete [ 2194.402866] LustreError: 63142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.423585] LustreError: 63142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2194.485652] LustreError: 64088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228009 with bad export cookie 4173935695543983388 [ 2194.495446] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.500889] LustreError: 64088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2194.816638] Lustre: server umount lustre-MDT0001 complete [ 2201.965812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2205.591705] Lustre: DEBUG MARKER: files compared the same [ 2217.974904] Lustre: server umount lustre-OST0000 complete [ 2235.735104] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2239.508953] Key type lgssc unregistered [ 2239.789931] LNet: 70437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.794191] LNetError: 70437:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.843110] LNet: Removed LNI 192.168.201.115@tcp [ 2240.701702] Key type .llcrypt unregistered [ 2240.703827] Key type ._llcrypt unregistered [ 2266.051809] Key type ._llcrypt registered [ 2266.057546] Key type .llcrypt registered [ 2266.176648] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2267.757257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.984777] alg: No test for adler32 (adler32-zlib) [ 2269.364676] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 2269.642581] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2271.417917] Key type lgssc registered [ 2273.282955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2283.854943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2283.875692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.569812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.573510] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug +malloc all [ 2294.819942] LustreError: 71836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.843597] LustreError: 71836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2299.040336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.463878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2304.268179] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug +malloc all [ 2311.221461] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2314.897519] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2321.965705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2322.284413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.352870] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug +malloc all [ 2330.474815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2333.819244] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2344.932937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2344.941217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.968180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2345.975034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.975198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2345.975623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.378947] Lustre: server umount lustre-OST0000 complete [ 2355.679862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2355.691224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.712804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.715450] Lustre: Skipped 1 previous similar message [ 2358.794304] Lustre: server umount lustre-MDT0000 complete [ 2361.313994] LustreError: 71832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.340964] LustreError: 71832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2362.938789] LustreError: 71818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228178 with bad export cookie 10179374129632364957 [ 2362.955747] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2363.315544] Lustre: server umount lustre-MDT0001 complete [ 2375.099889] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2378.584236] Key type lgssc unregistered [ 2378.944868] LNet: 75012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.950429] LNetError: 75012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.972940] LNet: Removed LNI 192.168.201.115@tcp [ 2379.942230] Key type .llcrypt unregistered [ 2379.945342] Key type ._llcrypt unregistered [ 2411.090515] Key type ._llcrypt registered [ 2411.095317] Key type .llcrypt registered [ 2411.285698] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2412.743092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.883622] alg: No test for adler32 (adler32-zlib) [ 2414.160041] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 2414.329123] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2415.978470] Key type lgssc registered [ 2417.521126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2426.205938] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2426.215843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2441.760926] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2446.880619] LustreError: 76387:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 2473.567835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.412510] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2486.240392] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2496.061669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.167498] Lustre: *** cfs_fail_loc=706, val=0*** [ 2496.783416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.938643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2502.112160] LustreError: 76412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.142133] LustreError: 76412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2503.191878] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2507.240529] LustreError: 76413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.280111] LustreError: 77364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.299493] LustreError: 77364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2513.384227] LustreError: 77362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.403614] LustreError: 77362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2513.473981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.841442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2517.555750] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2524.518367] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2529.148326] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2532.833608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2532.839966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.847995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.370614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.379081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2534.383222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.920785] Lustre: server umount lustre-OST0000 complete [ 2544.611648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.629774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2544.633790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.633799] Lustre: Skipped 1 previous similar message [ 2544.638624] Lustre: Skipped 1 previous similar message [ 2549.730421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2549.736715] Lustre: Skipped 2 previous similar messages [ 2553.825300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.834426] Lustre: Skipped 1 previous similar message [ 2556.384357] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2556.651277] Lustre: server umount lustre-MDT0000 complete [ 2558.946751] LustreError: 77368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.999432] LustreError: 77368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2560.762093] LustreError: 76394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228375 with bad export cookie 1523456861887180515 [ 2560.764020] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.769341] LustreError: 76394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2561.154029] Lustre: server umount lustre-MDT0001 complete [ 2573.464989] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2576.935331] Key type lgssc unregistered [ 2577.270745] LNet: 79650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2577.275949] LNetError: 79650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2577.287191] LNet: Removed LNI 192.168.201.115@tcp [ 2578.205392] Key type .llcrypt unregistered [ 2578.206504] Key type ._llcrypt unregistered [ 2604.812260] Key type ._llcrypt registered [ 2604.815307] Key type .llcrypt registered [ 2604.946679] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2606.404688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2606.592592] alg: No test for adler32 (adler32-zlib) [ 2607.861191] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 2608.314957] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2610.097250] Key type lgssc registered [ 2611.435902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2620.551609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2620.569528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.117327] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2636.353207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.095183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2642.401801] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.455089] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2642.952657] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2647.520913] LustreError: 81726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.541686] LustreError: 81726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2652.647988] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.676213] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2652.903761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.245520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.287595] LustreError: 81727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.325619] LustreError: 81727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2660.824695] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2662.381595] LustreError: 81721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.413755] LustreError: 81721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2670.365677] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2670.384244] Lustre: MGS: Client 25f77767-7222-4245-96b7-df985758e0d9 (at 0@lo) reconnecting [ 2670.412478] LustreError: 81737:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2670.694126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.814249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2676.199408] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2696.163751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.168394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2696.199744] Lustre: Skipped 1 previous similar message [ 2698.641456] Lustre: server umount lustre-OST0000 complete [ 2706.404584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2706.411173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2706.431126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2706.431135] Lustre: Skipped 1 previous similar message [ 2711.526395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2711.532512] Lustre: Skipped 2 previous similar messages [ 2716.641026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.651634] Lustre: Skipped 1 previous similar message [ 2717.157221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2717.374413] Lustre: server umount lustre-MDT0000 complete [ 2721.642767] Lustre: server umount lustre-MDT0001 complete [ 2725.432446] Lustre: server umount MGS complete [ 2743.379557] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:42:36 (1781228556) [ 2776.038731] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 2785.198481] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2819.325965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.697966] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2832.691609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.872424] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2832.914853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2832.976647] Lustre: lustre-MDT0001: new disk, initializing [ 2833.062057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2833.097416] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2833.104901] Lustre: 88814:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2833.114914] LustreError: 88814:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2837.653462] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2848.589541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.091341] Lustre: lustre-OST0000: new disk, initializing [ 2849.100450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2849.108150] Lustre: 89561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2849.230370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.120338] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2859.681461] Lustre: 90007:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000f2d7b0c3 x1867753374918144/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781228690 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2863.784745] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2872.634733] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2872.641757] Lustre: MGS: Client 865df2e3-fe8a-4dfd-9c53-f2bb6501943b (at 0@lo) reconnecting [ 2872.667780] Lustre: 88097:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2872.682517] LustreError: 88799:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2872.693263] LustreError: 88799:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2872.719369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2872.843900] Lustre: lustre-MDT0000: new disk, initializing [ 2873.078715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.103480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2873.212416] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2874.144647] Lustre: 90518:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000e42a07a9 x1867753374925312/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781228705 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2878.390766] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2879.967858] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2883.243395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2883.255290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2883.410931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2909.152200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.157767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.167752] Lustre: Skipped 1 previous similar message [ 2909.182032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.226635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.458174] Lustre: server umount lustre-OST0000 complete [ 2922.464147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2922.470472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.474024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2922.495259] Lustre: Skipped 1 previous similar message [ 2922.509164] Lustre: Skipped 3 previous similar messages [ 2925.216399] Lustre: server umount lustre-MDT0000 complete [ 2927.584996] LustreError: 88783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.600914] LustreError: 88783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2935.575235] Lustre: server umount lustre-MDT0001 complete [ 2939.209333] Lustre: server umount MGS complete [ 2950.371545] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:46:04 (1781228764) [ 2967.730920] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2970.500289] Key type lgssc unregistered [ 2970.825422] LNet: 92987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2970.842109] LNetError: 92987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2971.892841] LNet: Removed LNI 192.168.201.115@tcp [ 2972.832896] Key type .llcrypt unregistered [ 2972.837830] Key type ._llcrypt unregistered [ 2989.680175] Key type ._llcrypt registered [ 2989.684493] Key type .llcrypt registered [ 2989.804252] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2990.851580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2990.864396] alg: No test for adler32 (adler32-zlib) [ 2992.098392] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 2992.354388] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2994.087208] Key type lgssc registered [ 2995.161512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3002.506955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3002.546299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.227488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.615290] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3014.628940] LustreError: 94191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.663738] LustreError: 94191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3016.595777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3016.884793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3020.549505] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3025.171820] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3028.742861] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3035.886097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3036.290699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.818340] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3047.788356] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3049.655820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3063.268186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3063.272863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.295499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.312633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.312929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3065.314421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.321064] Lustre: Skipped 1 previous similar message [ 3067.760712] Lustre: server umount lustre-OST0000 complete [ 3073.508864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3073.523258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3073.552250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.556085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.556919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.580523] Lustre: Skipped 1 previous similar message [ 3078.600209] Lustre: server umount lustre-MDT0000 complete [ 3080.672208] LustreError: 95712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.680223] LustreError: 95712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3082.220418] LustreError: 96452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228897 with bad export cookie 6658032374205121755 [ 3082.229309] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3082.505392] Lustre: server umount lustre-MDT0001 complete [ 3092.890828] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3096.409210] Key type lgssc unregistered [ 3096.778284] LNet: 97388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3096.783830] LNetError: 97388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3096.812775] LNet: Removed LNI 192.168.201.115@tcp [ 3097.734148] Key type .llcrypt unregistered [ 3097.737670] Key type ._llcrypt unregistered [ 3113.887795] Key type ._llcrypt registered [ 3113.894633] Key type .llcrypt registered [ 3114.030835] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3115.340196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3115.412987] alg: No test for adler32 (adler32-zlib) [ 3116.475522] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 3116.830498] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3118.551168] Key type lgssc registered [ 3119.477293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3124.494163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3124.497910] LustreError: 98448:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3124.535382] LustreError: 98459:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3124.538606] LustreError: 98448:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4962 [ 3124.541563] LustreError: 98448:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3124.548089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.551750] LustreError: 98448:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3126.053413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3134.491433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.993388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3140.668806] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3145.924967] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing lsmod [ 3149.051367] LustreError: 99615:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3149.121945] LustreError: 99631:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3149.126985] LustreError: 99615:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4930 [ 3149.135808] LustreError: 99615:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3149.143178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.145389] LustreError: 99615:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3149.542674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.610646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3155.424558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3155.436977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.460703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.961683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.966448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.988872] Lustre: Skipped 1 previous similar message [ 3157.014198] Lustre: Skipped 2 previous similar messages [ 3158.896348] Lustre: server umount lustre-MDT0000 complete [ 3160.099902] LustreError: 98497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.130199] LustreError: 98497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3163.107630] LustreError: 98477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228978 with bad export cookie 7299366626813433996 [ 3163.119960] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.120496] LustreError: 98477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3164.255930] LustreError: 98927:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b388c96a x1867753908622080/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3164.624306] Lustre: server umount lustre-MDT0001 complete [ 3178.713439] Lustre: server umount lustre-OST0000 complete [ 3187.563487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.163360] LustreError: 100696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3188.276993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3192.906141] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3193.317359] LustreError: 100697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.435892] LustreError: 100696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.250727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.566318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3206.817328] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3213.021193] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3217.429311] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3224.387817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.971727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.996489] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3234.222808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3238.358823] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3247.676471] Lustre: server umount lustre-OST0000 complete [ 3253.728390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3253.758072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.788488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.880418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.895015] Lustre: Skipped 3 previous similar messages [ 3261.134709] Lustre: server umount lustre-MDT0000 complete [ 3263.968725] Lustre: 97861:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229063/real 1781229063] req@00000000e61702d6 x1867753908649344/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781229079 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3264.006821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.036831] Lustre: Skipped 1 previous similar message [ 3264.049668] LustreError: 100697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.068497] LustreError: 100697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3264.996397] Lustre: 97862:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229064/real 1781229064] req@00000000f8c72f31 x1867753908650112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781229080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3265.217976] LustreError: 100677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229080 with bad export cookie 7299366626813434927 [ 3265.229913] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3265.242260] LustreError: 100677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3265.845030] Lustre: server umount lustre-MDT0001 complete [ 3278.123415] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3281.372344] Key type lgssc unregistered [ 3281.799297] LNet: 103872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3281.823672] LNetError: 103872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3281.864552] LNet: Removed LNI 192.168.201.115@tcp [ 3282.914062] Key type .llcrypt unregistered [ 3282.917608] Key type ._llcrypt unregistered [ 3309.565648] Key type ._llcrypt registered [ 3309.568898] Key type .llcrypt registered [ 3309.712176] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3310.939257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3311.059323] alg: No test for adler32 (adler32-zlib) [ 3312.178980] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 3312.486931] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3314.247183] Key type lgssc registered [ 3315.389448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3323.471258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3323.493497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.888935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.116892] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3335.149440] LustreError: 105270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3335.166605] LustreError: 105270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3338.635787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.011044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3343.435993] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3349.936100] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3354.164239] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3361.750362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.299522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3370.155048] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3370.560834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3376.307836] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3382.873373] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3387.018476] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3395.048818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3395.069433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.090064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.066082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3396.072617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.083771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.128459] Lustre: Skipped 1 previous similar message [ 3401.189656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.203214] Lustre: Skipped 1 previous similar message [ 3405.279549] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3405.473719] Lustre: server umount lustre-OST0000 complete [ 3410.912722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3410.922269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.941859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.546220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.558711] Lustre: Skipped 2 previous similar messages [ 3424.223208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3424.522518] Lustre: server umount lustre-MDT0000 complete [ 3426.796127] LustreError: 105271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.823978] LustreError: 105271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3428.756551] LustreError: 105252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229243 with bad export cookie 18321622508630559689 [ 3428.759650] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.772981] LustreError: 105252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3429.230480] Lustre: server umount lustre-MDT0001 complete [ 3440.601794] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3443.904862] Key type lgssc unregistered [ 3444.433871] LNet: 108557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3444.443794] LNetError: 108557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3444.465418] LNet: Removed LNI 192.168.201.115@tcp [ 3445.611882] Key type .llcrypt unregistered [ 3445.615061] Key type ._llcrypt unregistered [ 3464.158798] Key type ._llcrypt registered [ 3464.161391] Key type .llcrypt registered [ 3464.279389] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3466.214954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3466.281299] alg: No test for adler32 (adler32-zlib) [ 3467.566775] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 3467.838556] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3469.551431] Key type lgssc registered [ 3470.901723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3479.352641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3479.375035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.939967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3485.917174] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3491.304813] LustreError: 109763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.314177] LustreError: 109763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3494.716213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.270489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3500.313285] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3506.276774] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3510.785498] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3518.126207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3518.600163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.024644] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3526.793516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3530.459362] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3541.474414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3541.486847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.508796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.513986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.519838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.537087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3544.352958] Lustre: server umount lustre-OST0000 complete [ 3551.713215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3551.725282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.757371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.772222] Lustre: Skipped 1 previous similar message [ 3554.700404] Lustre: server umount lustre-MDT0000 complete [ 3557.859241] LustreError: 109763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.879348] LustreError: 109763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3558.632384] LustreError: 109742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229373 with bad export cookie 3940870106469602505 [ 3558.636901] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.639876] LustreError: 109742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3559.100574] Lustre: server umount lustre-MDT0001 complete [ 3570.323491] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3573.482382] Key type lgssc unregistered [ 3573.864485] LNet: 112939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3573.875996] LNetError: 112939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3573.895778] LNet: Removed LNI 192.168.201.115@tcp [ 3574.908851] Key type .llcrypt unregistered [ 3574.919839] Key type ._llcrypt unregistered [ 3600.471433] Key type ._llcrypt registered [ 3600.474474] Key type .llcrypt registered [ 3600.598505] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3601.170785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3601.323491] alg: No test for adler32 (adler32-zlib) [ 3603.008104] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 3603.442528] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3605.159168] Key type lgssc registered [ 3607.372626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3615.861778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3615.887993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.680577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.616718] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3626.984240] LustreError: 114339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3627.018057] LustreError: 114339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3631.416146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3631.752938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3635.483338] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3640.990820] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3645.068817] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3651.975987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3652.409170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.586879] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3662.666666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3664.136480] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3670.250666] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3674.119368] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3674.128110] Lustre: Skipped 2 previous similar messages [ 3678.425716] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3678.434140] Lustre: Skipped 2 previous similar messages [ 3680.373693] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3680.409940] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3687.520881] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3687.530576] Lustre: Skipped 2 previous similar messages [ 3689.589430] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.15@tcp 192.168.0.[2,10]@tcp [ 3696.497890] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.15/24@tcp in log lustre-MDT0000 [ 3696.505914] Lustre: Skipped 2 previous similar messages [ 3696.998357] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.15/24@tcp [ 3697.011243] Lustre: Skipped 1 previous similar message [ 3703.778076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3703.781109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3703.817683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3705.585372] Lustre: server umount lustre-OST0000 complete [ 3713.506118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3713.512186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.534451] Lustre: Skipped 1 previous similar message [ 3713.543208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3713.549950] Lustre: Skipped 1 previous similar message [ 3716.124545] Lustre: server umount lustre-MDT0000 complete [ 3719.138634] LustreError: 114339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3719.167801] LustreError: 114339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3720.328289] LustreError: 116627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229535 with bad export cookie 13157934698165094885 [ 3720.333021] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3720.339928] LustreError: 116627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3721.104823] Lustre: server umount lustre-MDT0001 complete [ 3733.207564] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3736.418818] Key type lgssc unregistered [ 3736.715557] LNet: 118627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3736.729656] LNetError: 118627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3736.748864] LNet: Removed LNI 192.168.201.115@tcp [ 3737.781559] Key type .llcrypt unregistered [ 3737.783717] Key type ._llcrypt unregistered [ 3766.332791] Key type ._llcrypt registered [ 3766.336380] Key type .llcrypt registered [ 3766.474767] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3768.384055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3768.502171] alg: No test for adler32 (adler32-zlib) [ 3769.522487] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 3769.801261] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3771.455914] Key type lgssc registered [ 3772.471982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3780.660866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3780.676618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.235515] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3782.269904] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.15/24@tcp [ 3782.274435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.704112] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3791.849485] LustreError: 120170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.887520] LustreError: 120170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3795.496421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.794712] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3795.834994] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.15/24@tcp [ 3795.892765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3800.701553] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3806.752731] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3811.620401] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3819.292059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3819.654840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3826.521103] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3826.912345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3833.836235] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3846.634931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3846.659829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.695986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3847.648776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3847.664678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3847.680456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3847.690622] Lustre: Skipped 1 previous similar message [ 3849.332014] Lustre: server umount lustre-OST0000 complete [ 3857.379485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3857.388548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3857.407381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3863.027787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3863.033375] Lustre: Skipped 1 previous similar message [ 3868.129289] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3868.130067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3868.142699] Lustre: Skipped 1 previous similar message [ 3868.299101] Lustre: server umount lustre-MDT0000 complete [ 3871.549639] LustreError: 120150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229686 with bad export cookie 11517923144418405812 [ 3871.555128] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3871.559386] LustreError: 120150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3872.132723] Lustre: server umount lustre-MDT0001 complete [ 3884.547788] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3887.605330] Key type lgssc unregistered [ 3888.033794] LNet: 123414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3888.038806] LNetError: 123414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3889.062668] LNet: Removed LNI 192.168.201.115@tcp [ 3890.108532] Key type .llcrypt unregistered [ 3890.120952] Key type ._llcrypt unregistered [ 3916.750659] Key type ._llcrypt registered [ 3916.756429] Key type .llcrypt registered [ 3916.915406] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 3934.074246] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3935.561132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3935.691238] alg: No test for adler32 (adler32-zlib) [ 3936.820574] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 3937.011061] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3938.687825] Key type lgssc registered [ 3940.262578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3980.582707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3980.635633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.098536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3982.188237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3982.351642] Lustre: lustre-MDT0000: new disk, initializing [ 3982.491148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3982.523818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3988.825452] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4002.023326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.166642] Lustre: 126785:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4002.329347] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4002.338136] Lustre: Skipped 1 previous similar message [ 4002.432905] Lustre: lustre-MDT0001: new disk, initializing [ 4002.531836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4002.605914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4002.638952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4008.090871] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4017.035659] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4021.670814] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4029.312619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4029.538373] Lustre: lustre-OST0000: new disk, initializing [ 4029.541440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4029.545499] Lustre: 128525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4029.632356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4037.218860] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4038.266462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4038.287565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4038.469568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4045.518748] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4051.326427] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4051.609491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4056.545597] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4056.762865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4059.110251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4059.114289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.123112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4059.143812] Lustre: Skipped 1 previous similar message [ 4064.229061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4064.233645] Lustre: Skipped 2 previous similar messages [ 4064.819230] Lustre: server umount lustre-OST0000 complete [ 4069.345526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4069.345982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.352855] LustreError: Skipped 1 previous similar message [ 4069.368556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4069.376232] Lustre: Skipped 1 previous similar message [ 4074.466274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.491340] Lustre: Skipped 1 previous similar message [ 4075.965452] Lustre: server umount lustre-MDT0000 complete [ 4080.339791] LustreError: 126775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229895 with bad export cookie 1022226884613432556 [ 4080.352630] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4080.704557] Lustre: server umount lustre-MDT0001 complete [ 4089.925439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.551357] LustreError: 130594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.580613] LustreError: 130594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4090.666512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.982215] LustreError: 130595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.203800] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4100.071655] LustreError: 130594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.187382] LustreError: 130595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.160574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.575101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4111.267163] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4116.973744] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4120.996984] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4129.175732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4129.656406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4135.875627] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4143.564714] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4148.861452] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4149.170582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4154.458815] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4154.666330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4165.407319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4165.585828] Lustre: lustre-OST0001: new disk, initializing [ 4165.593752] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4165.598139] Lustre: 133595:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4165.695216] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4170.962203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4170.976908] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4171.071563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4173.415568] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4182.347216] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4182.617232] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4187.580617] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4187.886360] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4192.328796] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ff9e024c000.ost_server_uuid 50 [ 4194.237242] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ff9e024c000.ost_server_uuid in FULL state after 0 sec [ 4205.811988] Lustre: server umount lustre-OST0001 complete [ 4207.072439] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4207.078087] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.087108] LustreError: Skipped 1 previous similar message [ 4207.099180] LustreError: 132190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.105328] Lustre: Skipped 2 previous similar messages [ 4207.119200] LustreError: 132190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4212.197039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.235402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4212.250320] Lustre: Skipped 1 previous similar message [ 4215.993869] Lustre: server umount lustre-OST0000 complete [ 4222.432960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.437071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4222.460378] Lustre: Skipped 2 previous similar messages [ 4222.475424] Lustre: Skipped 1 previous similar message [ 4227.553521] LustreError: 132208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.566674] LustreError: 132208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4227.970903] Lustre: server umount lustre-MDT0000 complete [ 4231.972733] LustreError: 130574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230047 with bad export cookie 1022226884613433438 [ 4231.975241] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4231.985923] LustreError: 130574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4232.388773] Lustre: server umount lustre-MDT0001 complete [ 4251.943489] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 4255.187885] Key type lgssc unregistered [ 4255.604330] LNet: 136434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4255.626178] LNetError: 136434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4255.653672] LNet: Removed LNI 192.168.201.115@tcp [ 4256.759499] Key type .llcrypt unregistered [ 4256.761563] Key type ._llcrypt unregistered [ 4304.026573] Key type ._llcrypt registered [ 4304.028062] Key type .llcrypt registered [ 4304.291114] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 4321.358748] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 4323.120770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4323.193645] alg: No test for adler32 (adler32-zlib) [ 4324.490948] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 4324.895139] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 4326.594060] Key type lgssc registered [ 4328.161201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4365.691804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4365.718868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4367.009504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4367.032171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4367.074446] Lustre: lustre-MDT0000: new disk, initializing [ 4367.127834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.137627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4371.261868] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4383.948735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.159538] Lustre: 140561:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4384.243432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4384.250515] Lustre: Skipped 1 previous similar message [ 4384.442357] Lustre: lustre-MDT0001: new disk, initializing [ 4384.713763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4384.806822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4384.825568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4390.630609] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4398.461270] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4402.187907] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4408.873116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4409.094552] Lustre: lustre-OST0000: new disk, initializing [ 4409.098963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4409.105718] Lustre: 142300:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4409.185374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.501689] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4419.123565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4419.133328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4419.291101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4423.033865] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4427.852197] Lustre: Failing over lustre-OST0000 [ 4428.014025] Lustre: server umount lustre-OST0000 complete [ 4446.175259] Lustre: 138785:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230245/real 1781230245] req@0000000053ba5c7b x1867755175312512/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781230261 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4446.175877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.193127] Lustre: 138785:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4446.230080] Lustre: Skipped 1 previous similar message [ 4447.873687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4448.304065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.327429] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4450.787615] Lustre: 138785:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230250/real 1781230250] req@0000000039a1d6bc x1867755175314048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781230266 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4454.690720] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4455.906648] Lustre: 138784:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230255/real 1781230255] req@000000001cd04969 x1867755175315200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781230271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4455.951187] Lustre: 138784:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4456.431249] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4456.552917] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4456.556278] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4459.461688] Lustre: Failing over lustre-MDT0000 [ 4459.703037] Lustre: server umount lustre-MDT0000 complete [ 4461.535907] Lustre: 138783:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230260/real 1781230260] req@00000000c0edf8d7 x1867755175316352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781230276 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4461.541623] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.591456] Lustre: 138783:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4461.596549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4461.603726] Lustre: Skipped 1 previous similar message [ 4466.685770] LustreError: 140568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.703991] LustreError: 140568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4471.777723] LustreError: 140573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.790344] LustreError: 140573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4476.898543] Lustre: 138784:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230276/real 1781230276] req@000000007873b1b6 x1867755175326080/t0(0) o400->MGC192.168.201.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781230292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4476.904296] LustreError: 141681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.943711] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.993837] LustreError: 141681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4479.268838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.017895] LustreError: 140567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.043860] LustreError: 140567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4487.141438] LustreError: 140568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.186949] LustreError: 140568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4487.507137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4487.570953] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4488.253708] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4492.814930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4492.817659] Lustre: Skipped 1 previous similar message [ 4492.894322] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4492.979500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4494.257407] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4506.884946] Lustre: server umount lustre-OST0000 complete [ 4513.259204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.272594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4513.276322] Lustre: Skipped 2 previous similar messages [ 4518.373724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4518.383911] Lustre: Skipped 2 previous similar messages [ 4522.752517] Lustre: server umount lustre-MDT0000 complete [ 4523.489129] LustreError: 140573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.498150] LustreError: 140573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4524.518847] Lustre: 138783:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230323/real 1781230323] req@00000000fddb1b1f x1867755175345152/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781230339 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4524.567774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.613369] LustreError: 140553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230342 with bad export cookie 7351038893847620329 [ 4527.615960] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4527.632076] LustreError: 140553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4528.155855] Lustre: server umount lustre-MDT0001 complete [ 4540.736791] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 4544.234704] Key type lgssc unregistered [ 4544.666230] LNet: 145794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4544.672777] LNetError: 145794:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4544.701665] LNet: Removed LNI 192.168.201.115@tcp [ 4545.643954] Key type .llcrypt unregistered [ 4545.647069] Key type ._llcrypt unregistered [ 4581.490188] Key type ._llcrypt registered [ 4581.492659] Key type .llcrypt registered [ 4581.615219] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 4582.943780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4582.980308] alg: No test for adler32 (adler32-zlib) [ 4584.404644] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 4584.697260] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 4586.431288] Key type lgssc registered [ 4587.647840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4596.154695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4596.175500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.759656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.005700] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4606.956079] LustreError: 147301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.988212] LustreError: 147301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4609.887736] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4612.064668] LustreError: 147300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.001723] LustreError: 147329:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4614.019554] LustreError: 147329:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4614.027065] LustreError: 147329:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4614.273257] Lustre: server umount lustre-MDT0000 complete [ 4631.650960] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 4646.150735] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 4684.947770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.383968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4685.447825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4685.555914] Lustre: lustre-MDT0000: new disk, initializing [ 4685.642769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4685.662897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4690.440585] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4700.637454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4700.758551] Lustre: 150488:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4700.826809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4700.831026] Lustre: Skipped 1 previous similar message [ 4700.937307] Lustre: lustre-MDT0001: new disk, initializing [ 4701.066725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4701.112133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4701.127145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4705.773357] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4712.799443] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4717.110530] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4725.289192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4725.766258] Lustre: lustre-OST0000: new disk, initializing [ 4725.774922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4725.790807] Lustre: 152226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4725.974449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4731.023918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4731.052176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4731.182129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4732.441069] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4740.384913] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4744.490303] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4746.484218] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:16:00 (1781230560) [ 4972.000636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4972.011614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4972.023357] Lustre: Skipped 2 previous similar messages [ 4972.034952] Lustre: Skipped 1 previous similar message [ 4974.957946] Lustre: server umount lustre-MDT0000 complete [ 4977.123917] LustreError: 150498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4977.142761] LustreError: 150498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4979.179611] LustreError: 150479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230794 with bad export cookie 12102670884465938943 [ 4979.187708] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4979.193715] LustreError: 150479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4982.240934] LustreError: 150495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4982.242443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.299811] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4982.311169] Lustre: Skipped 1 previous similar message [ 4984.302421] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4985.563384] Lustre: server umount lustre-MDT0001 complete [ 5004.255148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5004.663375] Lustre: server umount lustre-OST0000 complete [ 5014.666285] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 5025.319533] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5066.178812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.365149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5066.383176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5066.432987] Lustre: lustre-MDT0000: new disk, initializing [ 5066.476633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.485952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5073.601323] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5085.470310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5085.592053] Lustre: 156737:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5085.678633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5085.681240] Lustre: Skipped 1 previous similar message [ 5085.767784] Lustre: lustre-MDT0001: new disk, initializing [ 5085.886394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5085.931570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5085.959481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5092.286313] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5103.542540] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5109.763629] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5120.442308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5120.747164] Lustre: lustre-OST0000: new disk, initializing [ 5120.755891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5120.796278] Lustre: 158480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5120.908440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5126.472271] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5128.776385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5128.786888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5128.937781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5135.468439] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5141.234289] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5141.522301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5146.894223] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5147.173989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5149.686886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5149.686915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.699672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5149.720552] Lustre: Skipped 1 previous similar message [ 5154.787624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5154.795996] Lustre: Skipped 2 previous similar messages [ 5155.848168] Lustre: server umount lustre-OST0000 complete [ 5162.979259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5162.987174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.005573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5166.564108] Lustre: server umount lustre-MDT0000 complete [ 5170.145470] LustreError: 156748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.178175] LustreError: 156748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5171.331188] LustreError: 156730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230986 with bad export cookie 12102670884466097773 [ 5171.352524] LustreError: 156730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5171.354549] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5172.008063] Lustre: server umount lustre-MDT0001 complete [ 5181.783581] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:23:15 (1781230995) [ 5200.434083] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 5211.076729] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5249.444770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.700300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5249.728992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5249.845904] Lustre: lustre-MDT0000: new disk, initializing [ 5249.955350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.969618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5254.675702] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5265.424750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.563897] Lustre: 163055:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5265.767351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5270.691374] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5279.379656] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5284.045971] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5292.179874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5292.779626] Lustre: lustre-OST0000: new disk, initializing [ 5292.789564] Lustre: Skipped 1 previous similar message [ 5292.800072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5292.809350] Lustre: Skipped 2 previous similar messages [ 5292.828254] Lustre: 164811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5300.655582] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5301.408873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5301.429570] Lustre: Skipped 1 previous similar message [ 5301.438651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5301.526686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5309.644981] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5322.209563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.233574] Lustre: Skipped 1 previous similar message [ 5322.238587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5322.244132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5322.246516] Lustre: Skipped 2 previous similar messages [ 5324.565425] Lustre: server umount lustre-OST0000 complete [ 5332.449357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.481316] Lustre: Skipped 2 previous similar messages [ 5332.484841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5335.362786] Lustre: server umount lustre-MDT0000 complete [ 5337.571132] LustreError: 163064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5337.600197] LustreError: 163064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5340.030729] LustreError: 163048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231155 with bad export cookie 12102670884466098655 [ 5340.050187] LustreError: 163048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5340.050717] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5340.340678] Lustre: server umount lustre-MDT0001 complete [ 5350.222900] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:26:04 (1781231164) [ 5368.235806] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 5379.940652] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5423.777227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.168163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5424.197910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5424.286847] Lustre: lustre-MDT0000: new disk, initializing [ 5424.449838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.468151] Lustre: Skipped 2 previous similar messages [ 5424.502164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5429.927577] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5443.048449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.236992] Lustre: 168997:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5443.557425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5449.091767] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5459.268795] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5463.999684] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5472.293812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5472.771183] Lustre: 170747:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5481.025460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5481.231071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5481.355746] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5490.768932] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5501.921352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5501.928503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.929718] LustreError: Skipped 1 previous similar message [ 5501.935481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5501.935495] Lustre: Skipped 2 previous similar messages [ 5501.980340] Lustre: Skipped 1 previous similar message [ 5506.403590] Lustre: server umount lustre-OST0000 complete [ 5517.133369] Lustre: server umount lustre-MDT0000 complete [ 5517.283047] LustreError: 171354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.298368] LustreError: 171354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5521.837623] LustreError: 168987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231337 with bad export cookie 12102670884466099740 [ 5521.846932] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5521.851237] LustreError: 168987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5522.153755] Lustre: server umount lustre-MDT0001 complete [ 5535.157116] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5538.644674] Key type lgssc unregistered [ 5538.943842] LNet: 172478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5538.950778] LNetError: 172478:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5540.025545] LNet: Removed LNI 192.168.201.115@tcp [ 5540.962745] Key type .llcrypt unregistered [ 5540.964165] Key type ._llcrypt unregistered [ 5567.171711] Key type ._llcrypt registered [ 5567.174208] Key type .llcrypt registered [ 5567.342692] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5568.801552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5568.842091] alg: No test for adler32 (adler32-zlib) [ 5569.917297] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 5570.160614] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 5571.839231] Key type lgssc registered [ 5573.264258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5580.662683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5580.678962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.203207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5586.485984] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5592.554555] LustreError: 173879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5592.566497] LustreError: 173879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5594.471618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5594.940877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5599.521543] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5606.076474] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5610.205323] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5618.715252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5619.149387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5626.389626] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5627.320231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5633.016207] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5652.961685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.965482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5652.980756] Lustre: Skipped 1 previous similar message [ 5652.981143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5653.013745] Lustre: Skipped 1 previous similar message [ 5657.625050] Lustre: server umount lustre-OST0000 complete [ 5663.204526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.227627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5663.235277] Lustre: Skipped 1 previous similar message [ 5668.324617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5668.336522] Lustre: Skipped 1 previous similar message [ 5668.639897] Lustre: server umount lustre-MDT0000 complete [ 5672.557941] LustreError: 173860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231487 with bad export cookie 17273207861516700899 [ 5672.569317] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5672.900271] Lustre: server umount lustre-MDT0001 complete [ 5686.438238] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5690.379704] Key type lgssc unregistered [ 5690.880303] LNet: 177080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5690.889407] LNetError: 177080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5690.907224] LNet: Removed LNI 192.168.201.115@tcp [ 5692.160501] Key type .llcrypt unregistered [ 5692.162088] Key type ._llcrypt unregistered [ 5716.848802] Key type ._llcrypt registered [ 5716.850450] Key type .llcrypt registered [ 5716.943071] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5718.001772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5718.212033] alg: No test for adler32 (adler32-zlib) [ 5719.340395] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 5719.584403] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 5721.272606] Key type lgssc registered [ 5722.344187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5729.619216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5729.649835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.370596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.329708] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5741.553392] LustreError: 178497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.579298] LustreError: 178497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5745.661943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.019396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5750.972319] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5756.722580] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5761.020779] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5768.055279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5768.368188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5774.187201] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5776.638325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5779.798820] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5787.117783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5787.130209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.146077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5787.153714] Lustre: Skipped 1 previous similar message [ 5787.741990] Lustre: lustre-OST0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 5792.241734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5792.253279] Lustre: Skipped 1 previous similar message [ 5792.822206] LustreError: 180089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.044429] Lustre: server umount lustre-OST0000 complete [ 5798.147811] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8ff9d190a800.ost_server_uuid 50 [ 5799.775264] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5817.058844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.067868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5817.083190] Lustre: Skipped 2 previous similar messages [ 5817.103084] Lustre: Skipped 1 previous similar message [ 5821.601889] Lustre: server umount lustre-MDT0000 complete [ 5822.435076] LustreError: 179208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.454679] LustreError: 179208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5825.283031] LustreError: 178479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231640 with bad export cookie 4865454968543242824 [ 5825.285956] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5825.290784] LustreError: 178479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5825.532814] Lustre: server umount lustre-MDT0001 complete [ 5834.356823] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:34:08 (1781231648) [ 5841.238788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5841.613390] LustreError: 181976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5841.666794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.361801] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5847.013946] LustreError: 181977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.307688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.756537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5860.588681] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5866.210333] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5870.689900] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5878.743507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.139583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5886.121085] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5886.321123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5892.003548] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5899.761946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5900.052161] Lustre: lustre-OST0001: new disk, initializing [ 5900.055922] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5900.065050] Lustre: 184478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5900.189449] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5906.607638] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5909.053806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5909.062894] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5909.158643] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5914.978796] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5919.720159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.731822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5919.756661] Lustre: Skipped 1 previous similar message [ 5924.363146] Lustre: server umount lustre-OST0000 complete [ 5924.848175] LustreError: 183572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.867740] LustreError: 183572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5931.956633] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5932.180908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5936.575201] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5936.776324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5941.305883] LustreError: 184486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5941.329947] LustreError: 184486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5954.800649] Lustre: server umount lustre-OST0001 complete [ 5960.672393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.699397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5960.716596] Lustre: Skipped 3 previous similar messages [ 5970.164592] Lustre: server umount lustre-MDT0000 complete [ 5970.914470] Lustre: 177809:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231770/real 1781231770] req@0000000018bf470d x1867756637088384/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781231786 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5970.954058] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.976685] Lustre: Skipped 1 previous similar message [ 5975.255182] LustreError: 181956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231790 with bad export cookie 4865454968543243895 [ 5975.259759] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5975.270097] LustreError: 181956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5976.036113] Lustre: server umount lustre-MDT0001 complete [ 6002.160024] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:36:56 (1781231816) [ 6008.833562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6009.170495] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6009.205852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6009.509351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.662149] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6025.390990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.501807] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6025.538706] Lustre: 187777:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6030.510348] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6037.815827] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6042.516781] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6049.729203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6049.929274] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6057.017546] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6059.662695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6062.763889] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6070.202451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6070.318102] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6076.215544] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6077.826410] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6084.151787] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6089.809816] Lustre: lustre-OST0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 6089.819000] Lustre: Skipped 2 previous similar messages [ 6090.208961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.223383] LustreError: Skipped 1 previous similar message [ 6090.226106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.948736] Lustre: server umount lustre-OST0000 complete [ 6094.897560] LustreError: 189396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.944465] LustreError: 189396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6112.225342] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6112.229866] LustreError: Skipped 1 previous similar message [ 6112.234498] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6112.242834] Lustre: Skipped 1 previous similar message [ 6116.463658] Lustre: server umount lustre-OST0001 complete [ 6122.476374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6122.484395] Lustre: Skipped 6 previous similar messages [ 6127.417458] Lustre: server umount lustre-MDT0000 complete [ 6130.762884] LustreError: 187768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231945 with bad export cookie 4865454968543245302 [ 6130.771202] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6130.776699] LustreError: 187768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6131.211317] Lustre: server umount lustre-MDT0001 complete [ 6159.591996] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:39:33 (1781231973) [ 6179.722900] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 6194.080964] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6235.569497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.740504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6235.815678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6236.002738] Lustre: lustre-MDT0000: new disk, initializing [ 6236.162932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6236.172375] Lustre: Skipped 3 previous similar messages [ 6236.209382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6240.605993] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6254.000188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.139035] Lustre: 195386:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6254.290529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6254.293156] Lustre: Skipped 1 previous similar message [ 6254.359226] Lustre: lustre-MDT0001: new disk, initializing [ 6254.537309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6254.546965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6259.817848] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6269.513297] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6273.686766] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6280.547320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6280.964608] Lustre: lustre-OST0000: new disk, initializing [ 6280.976074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6280.985328] Lustre: 197127:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6287.972590] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6289.593216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6289.608220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6289.768303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6296.843833] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6302.245454] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6302.501571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6307.650980] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6307.834332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6310.368711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6310.379359] LustreError: Skipped 1 previous similar message [ 6310.383303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.393877] Lustre: Skipped 3 previous similar messages [ 6310.399798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6316.269909] Lustre: server umount lustre-OST0000 complete [ 6327.223214] Lustre: server umount lustre-MDT0000 complete [ 6330.849895] LustreError: 195392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.858140] LustreError: 195392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6331.128648] LustreError: 195378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232146 with bad export cookie 4865454968543247087 [ 6331.137696] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6331.144662] LustreError: 195378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6332.078369] Lustre: server umount lustre-MDT0001 complete [ 6342.412807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.827294] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6357.446711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.375810] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6370.045347] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6375.351966] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6383.749430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6384.349698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.359457] Lustre: Skipped 4 previous similar messages [ 6392.717239] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6397.976703] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6401.462766] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6401.761971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6407.712207] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6408.006246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6413.282596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6413.289173] LustreError: Skipped 1 previous similar message [ 6413.290159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6413.314247] Lustre: Skipped 4 previous similar messages [ 6416.706427] Lustre: server umount lustre-OST0000 complete [ 6423.194313] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6427.913811] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6457.798682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6465.164596] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6472.329446] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6476.361224] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6477.839600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.854907] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.863714] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6477.876593] Lustre: Skipped 1 previous similar message [ 6482.923363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6482.933384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6482.935481] LustreError: Skipped 1 previous similar message [ 6482.953366] Lustre: Skipped 10 previous similar messages [ 6487.528354] Lustre: server umount lustre-OST0000 complete [ 6498.573669] Lustre: server umount lustre-MDT0000 complete [ 6502.515700] LustreError: 199178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232317 with bad export cookie 4865454968543247969 [ 6502.525595] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6502.531944] LustreError: 199178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6503.057643] Lustre: server umount lustre-MDT0001 complete [ 6511.986731] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:45:26 (1781232326) [ 6518.808769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.191655] LustreError: 204645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6519.211728] LustreError: 204645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6524.320216] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6533.755777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.629300] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6545.038120] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6548.422958] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6555.520740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.096964] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6568.901852] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6573.313208] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6573.558823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6578.641612] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6578.845281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6585.597426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6585.804552] Lustre: lustre-OST0001: new disk, initializing [ 6585.806973] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6585.811389] Lustre: 207642:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6592.678924] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6593.614057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6593.638484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6593.775264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6599.626986] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6604.115372] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6604.334994] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6609.212452] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6609.530313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6614.496480] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6614.506950] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6614.528898] Lustre: Skipped 5 previous similar messages [ 6617.583760] Lustre: server umount lustre-OST0001 complete [ 6623.699239] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6623.926839] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6629.004354] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6629.342665] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6658.397164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6658.631531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6658.640798] Lustre: Skipped 5 previous similar messages [ 6663.671909] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6663.690993] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6663.697799] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6663.715575] Lustre: Skipped 1 previous similar message [ 6664.914450] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6670.686234] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6672.315394] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6679.025414] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6679.030273] LustreError: Skipped 1 previous similar message [ 6680.862293] Lustre: server umount lustre-OST0001 complete [ 6704.361865] Lustre: server umount lustre-OST0000 complete [ 6715.750963] Lustre: server umount lustre-MDT0000 complete [ 6720.414732] LustreError: 204626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232535 with bad export cookie 4865454968543248935 [ 6720.419533] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6720.423959] LustreError: 204626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6727.014677] Lustre: server umount lustre-MDT0001 complete [ 6753.076230] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:49:27 (1781232567) [ 6759.437041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.872439] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6759.922583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6764.607958] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6772.601117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.682948] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6772.720952] Lustre: 212597:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6777.555634] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6783.342790] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6786.939694] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6794.688429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6794.960911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6802.616548] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6808.581978] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6818.355822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6818.456462] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6826.326854] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6832.726277] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6834.887489] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6840.550477] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6840.801115] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6846.047717] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6846.324107] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6850.855611] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ff9d0de4000.ost_server_uuid 50 [ 6852.503294] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ff9d0de4000.ost_server_uuid in IDLE state after 0 sec [ 6853.880301] Lustre: Permanently deactivating lustre-OST0001 [ 6853.921552] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6858.638525] Lustre: Permanently reactivating lustre-OST0001 [ 6858.660523] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6858.666837] Lustre: Skipped 2 previous similar messages [ 6866.917603] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6866.917744] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6866.944421] Lustre: Skipped 16 previous similar messages [ 6871.188949] Lustre: server umount lustre-OST0001 complete [ 6872.066073] LustreError: 216580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6872.102896] LustreError: 216580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 36 previous similar messages [ 6877.158234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6877.202658] Lustre: Skipped 9 previous similar messages [ 6882.751669] Lustre: server umount lustre-OST0000 complete [ 6894.387872] Lustre: server umount lustre-MDT0000 complete [ 6898.641955] LustreError: 212588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232713 with bad export cookie 4865454968543250223 [ 6898.657879] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6898.658760] LustreError: 212588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6899.183373] Lustre: server umount lustre-MDT0001 complete [ 6927.877611] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:52:21 (1781232741) [ 6936.804693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.173547] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6937.238164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6937.250521] Lustre: Skipped 2 previous similar messages [ 6942.356448] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6951.958907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6952.039439] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6952.076883] Lustre: 218711:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6956.987745] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6963.048615] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6966.911172] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6972.769970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6972.925627] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6972.946471] Lustre: Permanently deactivating lustre-OST0000 [ 6972.954709] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6979.965877] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6983.226413] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6985.736826] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6993.180617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6993.304496] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7000.393995] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7003.338522] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7006.701489] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7008.526229] Lustre: Permanently reactivating lustre-OST0000 [ 7008.569903] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7008.577641] Lustre: Skipped 2 previous similar messages [ 7009.778060] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7009.782902] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7009.803818] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7009.843664] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7009.869486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7024.120540] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7024.139297] LustreError: Skipped 2 previous similar messages [ 7028.780987] Lustre: server umount lustre-OST0001 complete [ 7047.135269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7047.443131] Lustre: server umount lustre-OST0000 complete [ 7066.079647] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7066.602868] Lustre: server umount lustre-MDT0000 complete [ 7070.454621] LustreError: 220353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232885 with bad export cookie 4865454968543251903 [ 7070.465550] LustreError: 220353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7070.468197] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7071.294490] Lustre: server umount lustre-MDT0001 complete [ 7081.738467] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7084.444885] Key type lgssc unregistered [ 7084.805603] LNet: 223123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7084.818508] LNetError: 223123:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7084.855832] LNet: Removed LNI 192.168.201.115@tcp [ 7085.422181] Key type .llcrypt unregistered [ 7085.423291] Key type ._llcrypt unregistered [ 7108.938382] Key type ._llcrypt registered [ 7108.941371] Key type .llcrypt registered [ 7109.116835] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7110.647547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7110.701819] alg: No test for adler32 (adler32-zlib) [ 7111.837116] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 7112.111888] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7113.751222] Key type lgssc registered [ 7115.300326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7126.648272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7126.689428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.495200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7133.600113] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7138.793751] LustreError: 224572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7138.804532] LustreError: 224572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7143.189061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.291471] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7143.325661] Lustre: 224563:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7143.347329] Lustre: Permanently deactivating lustre-MDT0001 [ 7143.373297] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7143.806021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7148.687340] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7150.111069] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7155.681554] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7160.398083] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7168.255074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7168.437809] LustreError: 226183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7168.666201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7173.809321] LustreError: 226185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.853200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7173.866331] LustreError: 226185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7173.870965] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7175.135836] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7179.251906] LustreError: 226183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.280020] LustreError: 226183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7181.060120] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7184.383840] LustreError: 226463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.405684] LustreError: 226463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7188.662357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7188.893882] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7194.097843] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7194.104244] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7196.143122] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7202.285364] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7206.470773] Lustre: Permanently reactivating lustre-MDT0001 [ 7206.500576] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7206.509961] Lustre: Skipped 2 previous similar messages [ 7208.870968] Lustre: lustre-MDT0001: Client d707b0cf-6599-4412-9b9f-009a8d199cbd (at 192.168.201.15@tcp) reconnecting [ 7213.311515] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7214.717400] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7214.734808] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7214.741652] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7214.759378] LustreError: 227795:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7214.760099] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7214.769244] LustreError: 227795:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7214.769385] LustreError: 227795:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.115@tcp: cfg command failed: rc = -108 [ 7214.774194] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7214.774194] [ 7214.800495] LustreError: 224565:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7215.149114] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7221.763816] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7221.961315] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7227.413585] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7227.575691] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7230.342228] Lustre: Permanently deactivating lustre-MDT0001 [ 7230.380137] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7230.384050] Lustre: Skipped 2 previous similar messages [ 7236.130156] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7245.280109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.292058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7245.295282] Lustre: Skipped 3 previous similar messages [ 7245.300112] Lustre: Skipped 2 previous similar messages [ 7246.896429] Lustre: server umount lustre-MDT0000 complete [ 7250.405337] LustreError: 224568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7250.431696] LustreError: 224568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7251.177580] LustreError: 224555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233066 with bad export cookie 4047451680379581654 [ 7251.187152] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7251.196314] LustreError: 224555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7252.023634] Lustre: server umount lustre-MDT0001 complete [ 7258.537208] Lustre: server umount lustre-OST0000 complete [ 7269.759213] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:58:03 (1781233083) [ 7280.760741] Lustre: server umount lustre-OST0001 complete [ 7288.041908] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 7298.862625] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7339.131715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7339.442346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7339.449073] Lustre: Skipped 2 previous similar messages [ 7339.515963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7339.603446] Lustre: lustre-MDT0000: new disk, initializing [ 7339.673976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.690295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7344.962922] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7356.151075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.342386] Lustre: 232215:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7356.401759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7356.405795] Lustre: Skipped 1 previous similar message [ 7356.498340] Lustre: lustre-MDT0001: new disk, initializing [ 7356.633597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7356.702260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7356.723421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7361.782202] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7370.301949] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7374.226419] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7381.161882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7381.487867] Lustre: lustre-OST0000: new disk, initializing [ 7381.495898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7381.513505] Lustre: 233956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7381.600965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.188823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7387.206246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7387.410888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7389.587928] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7398.451644] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7400.922636] LustreError: 234169:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7408.436989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7408.606859] Lustre: lustre-OST0001: new disk, initializing [ 7408.611819] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7408.622619] Lustre: 234957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7411.015262] LustreError: 234169:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7411.047833] LustreError: 232222:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7415.485349] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7418.001348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7418.027625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7418.177688] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7421.153111] LustreError: 232222:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7423.389533] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7425.024764] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7428.583810] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7428.586557] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.614210] LustreError: Skipped 1 previous similar message [ 7428.649573] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7428.663367] Lustre: Skipped 2 previous similar messages [ 7433.381518] Lustre: server umount lustre-OST0001 complete [ 7433.722103] LustreError: 234217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7433.755478] LustreError: 234217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7454.182405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.183088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7454.188599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7454.188610] Lustre: Skipped 1 previous similar message [ 7454.251260] Lustre: Skipped 2 previous similar messages [ 7457.256790] Lustre: server umount lustre-OST0000 complete [ 7464.421634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7464.423542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7464.423714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7464.423718] Lustre: Skipped 1 previous similar message [ 7464.451221] Lustre: Skipped 1 previous similar message [ 7464.499574] LustreError: Skipped 1 previous similar message [ 7469.537060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7469.553782] Lustre: Skipped 2 previous similar messages [ 7475.167170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7475.420924] Lustre: server umount lustre-MDT0000 complete [ 7478.974367] LustreError: 232206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233294 with bad export cookie 4047451680379584475 [ 7478.975619] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7478.981994] LustreError: 232206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7485.317666] Lustre: server umount lustre-MDT0001 complete [ 7495.950771] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7498.743384] Key type lgssc unregistered [ 7498.935862] LNet: 236871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7498.946300] LNetError: 236871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7498.965499] LNet: Removed LNI 192.168.201.115@tcp [ 7499.545482] Key type .llcrypt unregistered [ 7499.546994] Key type ._llcrypt unregistered [ 7543.437518] Key type ._llcrypt registered [ 7543.439695] Key type .llcrypt registered [ 7543.536439] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7545.135419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7545.189646] alg: No test for adler32 (adler32-zlib) [ 7546.418538] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 7546.662363] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7548.343202] Key type lgssc registered [ 7549.428290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7558.999100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7559.035725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.411370] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7560.448367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7560.712055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7565.781852] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7575.431167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7575.578957] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7575.664049] Lustre: 239065:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7576.178946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7580.390497] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7587.498617] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7591.909382] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7599.167607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7599.400217] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7599.691379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7605.275868] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7605.925081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7610.722315] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7667.169451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7667.184965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.210653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7667.691569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7667.717932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.732650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7669.619625] Lustre: server umount lustre-OST0000 complete [ 7674.079971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7687.411864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7687.990765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.998179] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7689.033708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7689.034567] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7689.064651] Lustre: Skipped 1 previous similar message [ 7694.604251] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7701.162830] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7745.520393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7745.521263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.531757] LustreError: Skipped 1 previous similar message [ 7745.532631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7745.574945] Lustre: Skipped 1 previous similar message [ 7747.811971] Lustre: server umount lustre-OST0000 complete [ 7755.251101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7755.254726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.282928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.288468] Lustre: Skipped 1 previous similar message [ 7758.149209] Lustre: server umount lustre-MDT0000 complete [ 7762.039571] LustreError: 239057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233577 with bad export cookie 12570849444481858905 [ 7762.044641] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.059776] LustreError: 239057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7762.389293] Lustre: server umount lustre-MDT0001 complete [ 7772.836060] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7775.776399] Key type lgssc unregistered [ 7776.092303] LNet: 244014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7776.104657] LNetError: 244014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7776.128530] LNet: Removed LNI 192.168.201.115@tcp [ 7776.850472] Key type .llcrypt unregistered [ 7776.852444] Key type ._llcrypt unregistered [ 7800.067544] Key type ._llcrypt registered [ 7800.069933] Key type .llcrypt registered [ 7800.152931] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7801.007124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7801.093295] alg: No test for adler32 (adler32-zlib) [ 7802.290148] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 7802.486065] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7804.135177] Key type lgssc registered [ 7805.327041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7812.282690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7812.309977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.861495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.151388] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7824.354513] LustreError: 245412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.364853] LustreError: 245412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7826.295328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.671603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7831.318894] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7837.365051] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7841.347873] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7848.396269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7848.728045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.245244] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7858.995778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7861.823388] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7886.816489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7886.821807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7886.834274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7889.889041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7889.898541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7889.908110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7889.933557] Lustre: Skipped 1 previous similar message [ 7892.013795] Lustre: server umount lustre-OST0000 complete [ 7898.594065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7898.604547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7898.623398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7905.249390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7905.257274] Lustre: Skipped 3 previous similar messages [ 7910.367975] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7910.369511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7910.399142] Lustre: Skipped 1 previous similar message [ 7910.518235] Lustre: server umount lustre-MDT0000 complete [ 7914.270430] LustreError: 245393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233729 with bad export cookie 16552801986208578466 [ 7914.282930] LustreError: 245393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7914.289274] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7914.600467] Lustre: server umount lustre-MDT0001 complete [ 7924.425996] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7927.268615] Key type lgssc unregistered [ 7927.598855] LNet: 249364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7927.606726] LNetError: 249364:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7927.629104] LNet: Removed LNI 192.168.201.115@tcp [ 7928.456483] Key type .llcrypt unregistered [ 7928.458086] Key type ._llcrypt unregistered [ 7944.796245] Key type ._llcrypt registered [ 7944.798128] Key type .llcrypt registered [ 7944.965159] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7945.807386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7945.824421] alg: No test for adler32 (adler32-zlib) [ 7946.779912] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 7946.922596] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7948.640955] Key type lgssc registered [ 7949.798728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7957.633557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7957.652182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.253926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7965.830576] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7969.769625] LustreError: 250576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7969.800807] LustreError: 250576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7974.893104] LustreError: 250577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7975.648820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.157418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7980.711632] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7986.554660] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7990.022786] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7996.888752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7997.187496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8004.415175] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8004.451312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8010.672540] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8025.061168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8025.073915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8025.077821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8025.086299] Lustre: Skipped 1 previous similar message [ 8029.754485] Lustre: server umount lustre-OST0000 complete [ 8035.297513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8035.309354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8035.332270] Lustre: Skipped 1 previous similar message [ 8039.908847] Lustre: server umount lustre-MDT0000 complete [ 8040.419399] LustreError: 250576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8040.443219] LustreError: 250576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8043.550366] LustreError: 252961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233858 with bad export cookie 2176028994740068367 [ 8043.557730] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8043.578373] LustreError: 252961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8044.335513] Lustre: server umount lustre-MDT0001 complete [ 8056.573397] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8059.438660] Key type lgssc unregistered [ 8059.661235] LNet: 253897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8059.672131] LNetError: 253897:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8059.683195] LNet: Removed LNI 192.168.201.115@tcp [ 8060.457351] Key type .llcrypt unregistered [ 8060.459929] Key type ._llcrypt unregistered [ 8083.147315] Key type ._llcrypt registered [ 8083.151888] Key type .llcrypt registered [ 8083.252375] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8084.288258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8084.309956] alg: No test for adler32 (adler32-zlib) [ 8085.471230] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 8085.728782] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 8087.383138] Key type lgssc registered [ 8088.611614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8096.051546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8096.072267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.666235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8101.311533] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8106.989147] LustreError: 255290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8107.018435] LustreError: 255290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8111.263634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.876367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8117.351710] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8124.831905] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8129.338608] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8136.842751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8137.286722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8142.402293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8144.725861] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8150.518203] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8178.158068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8178.171057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.194680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.244965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.246993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.256913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8184.375886] Lustre: server umount lustre-OST0000 complete [ 8188.898723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8188.914740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8188.942491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8188.951888] Lustre: Skipped 1 previous similar message [ 8192.483882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8192.490624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.258949] Lustre: server umount lustre-MDT0000 complete [ 8197.614358] LustreError: 255286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8197.654273] LustreError: 255286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8199.434559] LustreError: 255272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234014 with bad export cookie 880253365336364106 [ 8199.437690] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8199.447510] LustreError: 255272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8199.905921] Lustre: server umount lustre-MDT0001 complete [ 8211.322728] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8214.370445] Key type lgssc unregistered [ 8214.647986] LNet: 259241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8214.655582] LNetError: 259241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8214.669987] LNet: Removed LNI 192.168.201.115@tcp [ 8215.823745] Key type .llcrypt unregistered [ 8215.825874] Key type ._llcrypt unregistered [ 8239.383412] Key type ._llcrypt registered [ 8239.386691] Key type .llcrypt registered [ 8239.529903] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8241.365470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8241.391204] alg: No test for adler32 (adler32-zlib) [ 8242.691802] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 8242.884230] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 8244.591276] Key type lgssc registered [ 8246.151708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8255.339683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8255.354698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8257.103300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8261.816502] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8267.237398] LustreError: 260462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.256163] LustreError: 260462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8271.311264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.782831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8277.026409] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8283.609104] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8287.674676] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8294.890945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8295.251213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8301.357342] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8302.463666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8307.232592] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8322.016883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8322.026346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8322.045381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.046720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8323.051285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.058482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.095790] Lustre: Skipped 1 previous similar message [ 8326.664796] Lustre: server umount lustre-OST0000 complete [ 8333.282921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8333.295260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8333.297167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8333.302501] Lustre: Skipped 1 previous similar message [ 8337.023394] Lustre: server umount lustre-MDT0000 complete [ 8338.402397] LustreError: 260467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8338.428116] LustreError: 260467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8340.697229] LustreError: 260428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234155 with bad export cookie 15206471840565786576 [ 8340.699729] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8340.704557] LustreError: 260428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8341.197340] Lustre: server umount lustre-MDT0001 complete [ 8352.502827] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8355.549941] Key type lgssc unregistered [ 8355.997617] LNet: 263809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8356.005089] LNetError: 263809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8356.018183] LNet: Removed LNI 192.168.201.115@tcp [ 8356.989655] Key type .llcrypt unregistered [ 8356.994799] Key type ._llcrypt unregistered [ 8369.681440] Key type ._llcrypt registered [ 8369.683699] Key type .llcrypt registered [ 8369.811362] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8387.357160] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8403.503933] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8405.013127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8405.230061] alg: No test for adler32 (adler32-zlib) [ 8406.426635] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 8406.777591] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 8408.471187] Key type lgssc registered [ 8410.056476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8447.282797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8447.321265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8448.756650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8448.793893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8448.926827] Lustre: lustre-MDT0000: new disk, initializing [ 8449.056074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8449.075230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8453.443475] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8464.383190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.514593] Lustre: 267315:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8464.581865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8464.587054] Lustre: Skipped 1 previous similar message [ 8464.695820] Lustre: lustre-MDT0001: new disk, initializing [ 8464.783373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8464.822848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8464.830610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8469.374803] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8477.304613] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8480.744826] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8488.467484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8488.685737] Lustre: lustre-OST0000: new disk, initializing [ 8488.688621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8488.699092] Lustre: 269057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8488.869218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.733951] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8498.233467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8498.250049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8498.334653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8504.188637] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8509.345089] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8509.556174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8515.039910] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8515.319542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8519.137414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8519.140481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.149865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8519.167521] Lustre: Skipped 1 previous similar message [ 8523.548124] Lustre: server umount lustre-OST0000 complete [ 8529.381217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8529.388873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8529.389139] Lustre: Skipped 1 previous similar message [ 8529.426025] Lustre: Skipped 2 previous similar messages [ 8534.500168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8534.510201] Lustre: Skipped 1 previous similar message [ 8539.618361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8542.175139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8542.862397] Lustre: server umount lustre-MDT0000 complete [ 8546.323133] LustreError: 267305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234361 with bad export cookie 14087688271355625207 [ 8546.338835] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8546.894694] Lustre: server umount lustre-MDT0001 complete [ 8557.472817] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:19:31 (1781234371) [ 8564.302290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8564.643669] LustreError: 271320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8564.652099] LustreError: 271320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8564.703852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8569.164250] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8569.828403] LustreError: 271321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8573.921596] LustreError: 271320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8577.050840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8577.502960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8582.066400] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8588.073860] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8591.975656] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8599.364602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8599.766832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8606.861130] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8613.217970] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8625.632767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8625.639682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8625.651417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8625.653450] Lustre: Skipped 1 previous similar message [ 8625.675204] Lustre: Skipped 2 previous similar messages [ 8629.058915] Lustre: server umount lustre-OST0000 complete [ 8633.825459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8633.828318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.847365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.877916] Lustre: server umount lustre-MDT0000 complete [ 8641.002771] LustreError: 271315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8641.024545] LustreError: 271315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8644.192916] LustreError: 271301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234459 with bad export cookie 14087688271355626089 [ 8644.203674] LustreError: 271301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8644.207268] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8644.663604] Lustre: server umount lustre-MDT0001 complete [ 8656.546845] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8660.079228] Key type lgssc unregistered [ 8660.463642] LNet: 274501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8660.483409] LNetError: 274501:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8660.521644] LNet: Removed LNI 192.168.201.115@tcp [ 8661.328597] Key type .llcrypt unregistered [ 8661.330608] Key type ._llcrypt unregistered [ 8690.148265] Key type ._llcrypt registered [ 8690.150864] Key type .llcrypt registered [ 8690.271281] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8708.240110] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8709.671584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8709.733022] alg: No test for adler32 (adler32-zlib) [ 8710.791589] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 8711.052648] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 8712.743795] Key type lgssc registered [ 8713.879085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8758.282957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8758.346945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.725992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8759.794832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8759.896666] Lustre: lustre-MDT0000: new disk, initializing [ 8760.049962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8760.081346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8764.976951] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8775.651994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8775.806356] Lustre: 278295:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8775.902532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8775.905490] Lustre: Skipped 1 previous similar message [ 8776.032399] Lustre: lustre-MDT0001: new disk, initializing [ 8776.174620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8776.232249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8776.248765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8781.182361] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8787.944918] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8791.330833] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8797.747588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8798.012564] Lustre: lustre-OST2710: new disk, initializing [ 8798.014700] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8798.019204] Lustre: 280039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8798.149683] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8804.920637] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8806.964711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8806.971301] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8807.072599] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8812.844761] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8819.258333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8819.438755] Lustre: lustre-OST03e8: new disk, initializing [ 8819.444832] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8819.450782] Lustre: 280988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8819.570243] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8825.648449] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8828.024234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8828.045587] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8828.086640] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8830.414020] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8837.369951] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8837.594900] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8840.863930] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8841.155159] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8845.606707] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8845.835299] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8851.009803] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8851.289252] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8858.081259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8858.086566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8858.099752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8858.550880] Lustre: server umount lustre-MDT0000 complete [ 8859.106464] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8859.135044] Lustre: Skipped 1 previous similar message [ 8863.751759] LustreError: 278286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234678 with bad export cookie 8484251692415024978 [ 8863.767309] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8863.769680] LustreError: 278286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8864.264419] Lustre: server umount lustre-MDT0001 complete [ 8879.128771] Lustre: server umount lustre-OST2710 complete [ 8880.863124] Lustre: 276096:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234679/real 1781234679] req@0000000049f27a1e x1867759774399232/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781234695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8880.901083] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8880.920161] Lustre: Skipped 1 previous similar message [ 8883.027964] Lustre: server umount lustre-OST03e8 complete [ 8900.648121] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8910.534903] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8944.654848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8944.891284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8944.916838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8944.978163] Lustre: lustre-MDT0000: new disk, initializing [ 8945.065349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.087736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8949.303845] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8958.477813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8958.610635] Lustre: 286024:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8958.685270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8958.695098] Lustre: Skipped 1 previous similar message [ 8958.803348] Lustre: lustre-MDT0001: new disk, initializing [ 8958.898840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8958.936965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8958.950689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8963.668918] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8970.982245] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8974.129749] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8980.707669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8980.998877] Lustre: lustre-OST0000: new disk, initializing [ 8981.001718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8981.006919] Lustre: 287764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8981.086842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8986.931143] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8990.292813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8990.307476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8990.480583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8994.470690] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8999.501748] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8999.759257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9004.733758] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9005.021942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9011.174690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9011.179491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9011.185716] LustreError: Skipped 1 previous similar message [ 9011.207431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9011.217341] Lustre: Skipped 1 previous similar message [ 9016.289766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9016.298970] Lustre: Skipped 1 previous similar message [ 9021.408336] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9021.409957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9021.429927] Lustre: Skipped 1 previous similar message [ 9021.855248] Lustre: server umount lustre-OST0000 complete [ 9026.532065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.546337] Lustre: Skipped 1 previous similar message [ 9026.551665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9026.555690] Lustre: Skipped 1 previous similar message [ 9036.770903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9036.777514] Lustre: Skipped 3 previous similar messages [ 9040.353633] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9040.631056] Lustre: server umount lustre-MDT0000 complete [ 9041.893265] LustreError: 286030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9041.928701] LustreError: 286030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9045.081542] LustreError: 287763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234860 with bad export cookie 8484251692415026651 [ 9045.087259] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9045.096861] LustreError: 287763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9045.659506] Lustre: server umount lustre-MDT0001 complete [ 9055.701742] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:27:49 (1781234869) [ 9056.943887] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9058.719995] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:27:52 (1781234872) [ 9061.952976] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9092.187673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.365235] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9092.379932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9092.511494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9097.375598] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9106.577793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9106.684109] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9106.726776] Lustre: 291462:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9110.802867] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9132.022911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9132.407692] Lustre: Denying initial registration attempt from nid 192.168.201.115@tcp, specified as failover [ 9132.417158] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9132.431661] LustreError: 292882:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 9132.451380] LustreError: 292882:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9132.462213] LustreError: 292882:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9132.574709] Lustre: server umount lustre-OST0000 complete [ 9132.584266] LustreError: 292882:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9142.243303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9142.248496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9142.252487] Lustre: Skipped 2 previous similar messages [ 9142.261057] Lustre: Skipped 1 previous similar message [ 9146.133917] Lustre: server umount lustre-MDT0000 complete [ 9147.361525] LustreError: 291473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9147.399838] LustreError: 291473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9150.501440] LustreError: 291454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234965 with bad export cookie 8484251692415027540 [ 9150.528591] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9151.372863] Lustre: server umount lustre-MDT0001 complete [ 9163.936808] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 9167.300882] Key type lgssc unregistered [ 9167.833538] LNet: 293986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9167.848589] LNetError: 293986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9167.875616] LNet: Removed LNI 192.168.201.115@tcp [ 9168.974946] Key type .llcrypt unregistered [ 9168.978093] Key type ._llcrypt unregistered [ 9181.271945] Key type ._llcrypt registered [ 9181.274847] Key type .llcrypt registered [ 9181.430450] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9182.687882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9182.812067] alg: No test for adler32 (adler32-zlib) [ 9183.885516] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 9184.142316] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 9185.927238] Key type lgssc registered [ 9187.240222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9235.618570] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9243.746688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9243.761139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.204442] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9245.248449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9245.515313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9249.588677] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9258.249134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9258.349585] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9258.377853] Lustre: 296720:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9258.635322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9261.902290] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9267.210369] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9270.926268] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9290.541865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9290.788100] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9290.955709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.144760] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9303.029754] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9309.152675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9309.160062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.174310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9309.665593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9309.677503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.699156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9311.819310] Lustre: server umount lustre-OST0000 complete [ 9319.913039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9319.920153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9319.931511] Lustre: Skipped 1 previous similar message [ 9319.960233] Lustre: Skipped 1 previous similar message [ 9321.994371] Lustre: server umount lustre-MDT0000 complete [ 9326.194623] LustreError: 297445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235141 with bad export cookie 8877417959406762581 [ 9326.195817] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9326.206881] LustreError: 297445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9326.471061] Lustre: server umount lustre-MDT0001 complete [ 9339.030665] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 9342.212202] Key type lgssc unregistered [ 9342.532668] LNet: 299937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9342.543448] LNetError: 299937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9342.579800] LNet: Removed LNI 192.168.201.115@tcp [ 9343.532147] Key type .llcrypt unregistered [ 9343.533985] Key type ._llcrypt unregistered [ 9367.981784] Key type ._llcrypt registered [ 9367.984395] Key type .llcrypt registered [ 9368.087926] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9369.231955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9369.282129] alg: No test for adler32 (adler32-zlib) [ 9370.434868] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 9370.667296] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 9372.359199] Key type lgssc registered [ 9373.538831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9380.657244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9380.679239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.303247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9387.155376] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9392.615237] LustreError: 301344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9392.633119] LustreError: 301344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9394.732045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9395.145221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9400.221085] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9406.302437] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9410.992753] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9420.068434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9420.529720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9427.070064] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9432.806528] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9441.253926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9441.269792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9441.293969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9443.304236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9443.319254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9445.985053] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 9445.987806] Lustre: Skipped 2 previous similar messages [ 9447.482404] Lustre: server umount lustre-MDT0000 complete [ 9448.417424] LustreError: 303268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9448.424609] LustreError: 303268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9451.056728] LustreError: 301339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9451.776717] LustreError: 301326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235267 with bad export cookie 156287060423887937 [ 9451.784762] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9451.795388] LustreError: 301326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9453.536453] LustreError: 301340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9453.537540] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9453.566253] Lustre: Skipped 1 previous similar message [ 9453.569735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9461.293130] LustreError: 301339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9461.294051] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [ 9461.325360] LustreError: 301339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9461.350197] Lustre: Skipped 2 previous similar messages [ 9466.335312] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9466.747850] Lustre: server umount lustre-MDT0001 complete [ 9473.395323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9481.694565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9497.946067] LustreError: 302951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.961338] LustreError: 302951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9498.028032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9498.073272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9502.355881] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9511.192741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.563027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9517.215155] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9518.057580] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9523.941692] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9525.718818] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9530.259575] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9531.955299] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9538.539391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.540576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9538.553653] Lustre: Skipped 1 previous similar message [ 9538.561096] Lustre: Skipped 4 previous similar messages [ 9543.572548] Lustre: server umount lustre-OST0000 complete [ 9548.769863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9548.789111] Lustre: Skipped 1 previous similar message [ 9553.920985] LustreError: 304600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.952769] LustreError: 304600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9554.322585] Lustre: server umount lustre-MDT0000 complete [ 9558.673663] LustreError: 301326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235373 with bad export cookie 156287060423895588 [ 9558.677216] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9558.684974] LustreError: 301326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9559.081165] Lustre: server umount lustre-MDT0001 complete [ 9571.670734] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 9575.188936] Key type lgssc unregistered [ 9575.638403] LNet: 307033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9575.646738] LNetError: 307033:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9575.658924] LNet: Removed LNI 192.168.201.115@tcp [ 9576.684530] Key type .llcrypt unregistered [ 9576.685950] Key type ._llcrypt unregistered [ 9605.237270] Key type ._llcrypt registered [ 9605.246852] Key type .llcrypt registered [ 9605.406021] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9607.148415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9607.174883] alg: No test for adler32 (adler32-zlib) [ 9608.333867] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 9608.549758] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 9610.215174] Key type lgssc registered [ 9611.648293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9620.870544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9620.925619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.497865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9627.552748] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9632.739325] LustreError: 308436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9632.761441] LustreError: 308436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9636.400043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.787644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9641.143382] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9647.008767] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9650.697579] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9654.236085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9654.237639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9654.256891] Lustre: Skipped 1 previous similar message [ 9654.272965] Lustre: Skipped 1 previous similar message [ 9659.361632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.367289] Lustre: Skipped 1 previous similar message [ 9660.845341] Lustre: server umount lustre-MDT0000 complete [ 9661.411940] LustreError: 308432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9661.451132] LustreError: 308432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9665.061283] LustreError: 308416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235480 with bad export cookie 3279482571144566968 [ 9665.085394] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9665.467625] Lustre: server umount lustre-MDT0001 complete [ 9681.323155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9681.543896] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9681.595857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9681.828338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9686.187279] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9693.777092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9693.902157] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9693.953572] Lustre: 310798:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9694.256970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9699.114433] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9706.872321] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9711.096874] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9720.772580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9721.086457] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9721.094059] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9721.102499] LustreError: 312454:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9721.112064] LustreError: 312454:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9721.132434] LustreError: 312454:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9721.225857] Lustre: server umount lustre-OST0000 complete [ 9721.228252] LustreError: 312454:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9731.315071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9731.491275] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9731.688130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9737.389303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9737.864874] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9742.952232] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9750.992359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9751.124384] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9751.207809] Lustre: lustre-OST0001: new disk, initializing [ 9751.211226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9751.215822] Lustre: 313883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9751.342772] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9759.653689] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9760.998503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9761.019833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9761.241956] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9768.216372] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9776.610823] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9776.627401] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9776.633102] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9776.633647] Lustre: Skipped 1 previous similar message [ 9778.413776] Lustre: server umount lustre-OST0001 complete [ 9781.750268] LustreError: 313233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9781.793597] LustreError: 313233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9786.859871] LustreError: 313233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.861084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.864569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9786.870162] LustreError: 313233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9786.914112] Lustre: Skipped 2 previous similar messages [ 9790.412903] Lustre: server umount lustre-OST0000 complete [ 9796.578778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9796.582193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9796.619553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9796.625298] Lustre: Skipped 1 previous similar message [ 9802.188288] Lustre: server umount lustre-MDT0000 complete [ 9802.221791] LustreError: 310805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9802.272244] LustreError: 310805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9807.009639] LustreError: 310790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235622 with bad export cookie 3279482571144567521 [ 9807.021950] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9807.028961] LustreError: 310790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9807.899770] Lustre: server umount lustre-MDT0001 complete [ 9821.027762] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 9825.126025] Key type lgssc unregistered [ 9825.426646] LNet: 315713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9825.445953] LNetError: 315713:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9825.478263] LNet: Removed LNI 192.168.201.115@tcp [ 9826.661748] Key type .llcrypt unregistered [ 9826.666670] Key type ._llcrypt unregistered [ 9907.556690] Key type ._llcrypt registered [ 9907.559482] Key type .llcrypt registered [ 9907.714293] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 9924.095794] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9924.890187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9925.062064] alg: No test for adler32 (adler32-zlib) [ 9926.023213] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 9926.265898] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 9927.983181] Key type lgssc registered [ 9929.355237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9974.669274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9974.707769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.224869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9976.297153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9976.400323] Lustre: lustre-MDT0000: new disk, initializing [ 9976.551291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9976.573206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9982.941885] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9996.285309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9996.502991] Lustre: 320616:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9996.579932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9996.583517] Lustre: Skipped 1 previous similar message [ 9996.685954] Lustre: lustre-MDT0001: new disk, initializing [ 9996.854604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9996.905959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9996.924769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10003.209783] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10013.018878] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10018.829450] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10027.723440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10028.046583] Lustre: lustre-OST0000: new disk, initializing [10028.052557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10028.065771] Lustre: 322359:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10028.210114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10034.631645] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10036.832413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10036.840102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10036.945935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10043.712468] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10051.069284] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10051.545944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10058.046667] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10058.228148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10062.819266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10062.820783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10062.821884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10062.834173] LustreError: Skipped 1 previous similar message [10062.860505] Lustre: Skipped 1 previous similar message [10067.481920] Lustre: server umount lustre-OST0000 complete [10073.063622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10073.092442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10073.094834] Lustre: Skipped 1 previous similar message [10078.182857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10078.191758] Lustre: Skipped 1 previous similar message [10078.950393] Lustre: server umount lustre-MDT0000 complete [10083.685340] LustreError: 320609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235898 with bad export cookie 14320095102122732835 [10083.691318] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10083.705817] LustreError: 320609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10084.259871] Lustre: server umount lustre-MDT0001 complete [10093.743864] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:45:07 (1781235907) [10101.426906] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:45:15 (1781235915) [10110.341029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.684081] LustreError: 324969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.700995] LustreError: 324969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10110.753353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10114.462681] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10116.071488] LustreError: 324970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10120.178119] LustreError: 324969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.902209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.412225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10127.460456] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10133.010736] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10136.676039] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10142.812374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10143.214871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10149.382803] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10155.607715] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10163.939965] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10165.994198] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10168.165268] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10170.201471] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10172.182533] Lustre: Failing over lustre-MDT0000 [10172.654213] Lustre: server umount lustre-MDT0000 complete [10173.525739] LustreError: 324965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10174.444418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10174.453937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10174.483561] Lustre: Skipped 1 previous similar message [10178.617686] LustreError: 326580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10178.638036] LustreError: 326580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10188.850738] LustreError: 324964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.882942] LustreError: 324964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10192.289332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10192.351153] Lustre: 318840:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781235991/real 1781235991] req@00000000d6b787ee x1867761048426496/t0(0) o400->MGC192.168.201.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781236007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10192.381882] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10202.593250] LustreError: 318837:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c8a5fc58 x1867761048431872/t0(0) o250->MGC192.168.201.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10203.098137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10203.166723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10205.239574] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10208.233405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10208.314307] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10208.364732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10208.606661] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10217.497206] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10219.557336] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10223.720590] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10226.302331] Lustre: Failing over lustre-MDT0000 [10226.625483] Lustre: server umount lustre-MDT0000 complete [10228.721494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10228.723902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10228.735340] LustreError: 324965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10228.735353] LustreError: 324965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [10228.783570] Lustre: Skipped 4 previous similar messages [10244.064727] Lustre: 318838:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781236043/real 1781236043] req@00000000dc2e0161 x1867761048448896/t0(0) o400->MGC192.168.201.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781236059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10244.110511] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10247.697295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10254.307320] LustreError: 318837:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005b649eca x1867761048453632/t0(0) o250->MGC192.168.201.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10254.623767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10254.709077] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10257.464272] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10259.873978] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10259.956572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10259.969371] Lustre: Skipped 2 previous similar messages [10260.014943] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10260.080239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10267.838565] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10269.871397] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10280.419266] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10280.422303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10280.439422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10280.445295] Lustre: Skipped 1 previous similar message [10282.288502] Lustre: server umount lustre-MDT0000 complete [10285.542445] LustreError: 326580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.575325] LustreError: 326580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [10286.567355] LustreError: 324950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236101 with bad export cookie 14320095102122735544 [10286.569793] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10286.585645] LustreError: 324950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10287.002455] Lustre: server umount lustre-MDT0001 complete [10301.847531] Lustre: server umount lustre-OST0000 complete [10318.252488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10319.060273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10324.359749] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10333.677293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10340.039314] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10346.968996] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10351.822473] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10361.085064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10364.743782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10369.721736] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10375.901354] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10387.945471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [10387.955980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10387.980281] Lustre: Skipped 1 previous similar message [10387.990241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10388.001501] Lustre: Skipped 2 previous similar messages [10393.801222] Lustre: server umount lustre-OST0000 complete [10400.737084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10400.740420] LustreError: Skipped 1 previous similar message [10400.747473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10400.752162] Lustre: Skipped 2 previous similar messages [10404.452828] Lustre: server umount lustre-MDT0000 complete [10405.862359] LustreError: 330744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.882970] LustreError: 330744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10408.559251] LustreError: 330725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236223 with bad export cookie 14320095102122736286 [10408.574447] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10409.076933] Lustre: server umount lustre-MDT0001 complete [10421.484747] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [10424.411821] Key type lgssc unregistered [10424.677196] LNet: 333922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10424.681837] LNetError: 333922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10424.691221] LNet: Removed LNI 192.168.201.115@tcp [10425.684227] Key type .llcrypt unregistered [10425.689721] Key type ._llcrypt unregistered [10457.685694] Key type ._llcrypt registered [10457.687815] Key type .llcrypt registered [10457.802398] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10459.231535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10459.404705] alg: No test for adler32 (adler32-zlib) [10460.532474] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [10460.746194] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [10462.407675] Key type lgssc registered [10463.858766] Lustre: Echo OBD driver; http://www.lustre.org/ [10472.539920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10472.582437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.122692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10479.155985] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10484.707477] LustreError: 335419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.722305] LustreError: 335419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10488.233254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10488.668244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10493.497993] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10499.334084] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10503.765076] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10511.209357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10511.571723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10518.532408] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10519.731143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10524.852274] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10538.324278] Lustre: server umount lustre-MDT0000 complete [10540.001798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10540.013564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10540.026820] LustreError: 335420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10540.529311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10542.226071] LustreError: 335400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236357 with bad export cookie 14320619478232609233 [10542.231881] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10542.243604] LustreError: 335400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10542.700102] Lustre: server umount lustre-MDT0001 complete [10557.309989] Lustre: server umount lustre-OST0000 complete [10571.747806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10572.314217] LustreError: 338791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10572.329255] LustreError: 338791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10572.408745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10577.016596] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10577.909506] LustreError: 338792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.056774] LustreError: 338791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10585.617021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10585.963762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10590.245540] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10596.610987] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10600.640567] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10608.154041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10608.475962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10615.378552] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10617.769893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10622.776993] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10637.280397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10637.305745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10637.323915] Lustre: Skipped 1 previous similar message [10637.327454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10638.310373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10642.453068] Lustre: server umount lustre-MDT0000 complete [10643.431518] LustreError: 338792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10643.441619] LustreError: 338792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10645.698583] LustreError: 338773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236460 with bad export cookie 14320619478232610353 [10645.702984] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10645.708385] LustreError: 338773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10645.950251] Lustre: server umount lustre-MDT0001 complete [10659.135915] Lustre: server umount lustre-OST0000 complete [10677.619491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.001830] LustreError: 342363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.059758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10683.561672] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10692.926953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.515199] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10704.155973] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10708.344439] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10715.461508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10715.997548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10716.002613] Lustre: Skipped 1 previous similar message [10723.233954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10723.877195] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10733.977281] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10743.782337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10743.782856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10743.797434] LustreError: Skipped 1 previous similar message [10743.798334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10743.798339] Lustre: Skipped 2 previous similar messages [10743.858745] Lustre: Skipped 3 previous similar messages [10748.483915] Lustre: server umount lustre-OST0000 complete [10754.018452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10754.021164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10754.040628] Lustre: Skipped 1 previous similar message [10754.067264] Lustre: Skipped 2 previous similar messages [10759.137427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10759.155262] Lustre: Skipped 1 previous similar message [10767.329633] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10767.566301] Lustre: server umount lustre-MDT0000 complete [10769.377421] LustreError: 343327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.413394] LustreError: 343327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10771.650149] LustreError: 342344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236586 with bad export cookie 14320619478232611375 [10771.658112] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10771.661091] LustreError: 342344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10771.982984] Lustre: server umount lustre-MDT0001 complete [10783.493432] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [10786.193480] Key type lgssc unregistered [10786.482805] LNet: 345542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10786.488693] LNetError: 345542:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10787.563566] LNet: Removed LNI 192.168.201.115@tcp [10788.517566] Key type .llcrypt unregistered [10788.519786] Key type ._llcrypt unregistered [10813.886494] Key type ._llcrypt registered [10813.890339] Key type .llcrypt registered [10814.029268] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10815.596180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10815.749193] alg: No test for adler32 (adler32-zlib) [10816.819810] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [10817.046893] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [10818.711360] Key type lgssc registered [10820.048798] Lustre: Echo OBD driver; http://www.lustre.org/ [10827.600061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10827.629169] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10827.640908] LustreError: 346968:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10827.657745] LustreError: 346968:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10827.664094] LustreError: 346968:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10827.673862] LustreError: 346968:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10827.686225] LustreError: 346968:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10836.333255] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10836.337519] LustreError: 347365:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10836.359203] LustreError: 347365:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10836.363723] LustreError: 347365:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10836.368682] LustreError: 347365:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10836.377789] LustreError: 347365:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10854.959228] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [10857.845658] Key type lgssc unregistered [10858.239804] LNet: 348350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10858.250049] LNetError: 348350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10858.278489] LNet: Removed LNI 192.168.201.115@tcp [10859.527069] Key type .llcrypt unregistered [10859.528700] Key type ._llcrypt unregistered [10882.434043] Key type ._llcrypt registered [10882.435399] Key type .llcrypt registered [10882.599734] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [10899.848553] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10901.304488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10901.331470] alg: No test for adler32 (adler32-zlib) [10902.468520] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [10902.711283] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [10904.391889] Key type lgssc registered [10905.891251] Lustre: Echo OBD driver; http://www.lustre.org/ [10946.076420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10946.110459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10947.462585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10947.512146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10947.640156] Lustre: lustre-MDT0000: new disk, initializing [10947.778062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10947.826936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10952.495970] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10964.830061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10965.025454] Lustre: 351479:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10965.148165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10965.152582] Lustre: Skipped 1 previous similar message [10965.233688] Lustre: lustre-MDT0001: new disk, initializing [10965.335281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10965.376519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10965.402223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10971.016373] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10979.213364] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10983.050328] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10990.320963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10990.701566] Lustre: lustre-OST0000: new disk, initializing [10990.712667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10990.718391] Lustre: 353225:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10990.826156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10997.445052] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11000.427061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11000.435624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11000.601206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11005.941288] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11010.549926] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11010.813745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11015.987652] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11016.332464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11021.296691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11021.299084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11021.312315] LustreError: Skipped 1 previous similar message [11021.314955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11021.320710] Lustre: Skipped 1 previous similar message [11021.350430] Lustre: Skipped 1 previous similar message [11026.407311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11026.432278] Lustre: Skipped 1 previous similar message [11031.520914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11033.056701] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11033.238937] Lustre: server umount lustre-OST0000 complete [11041.761758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11041.769300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11041.778951] Lustre: Skipped 1 previous similar message [11041.790689] Lustre: Skipped 2 previous similar messages [11044.257670] Lustre: server umount lustre-MDT0000 complete [11048.473869] LustreError: 351470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236863 with bad export cookie 13828966451976871481 [11048.475841] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11048.488311] LustreError: 351470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11049.126899] Lustre: server umount lustre-MDT0001 complete [11057.984400] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11060.412562] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:01:13 (1781236873) [11066.313564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11066.790454] LustreError: 355549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11066.811151] LustreError: 355549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11066.849432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11071.993480] LustreError: 355550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11072.285567] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11077.091051] LustreError: 355549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.987099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.456982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11087.625975] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11096.537428] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11100.794252] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11107.926196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11108.274198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11114.084642] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11120.057201] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11128.204764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11128.380815] Lustre: lustre-OST0001: new disk, initializing [11128.384403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11128.389603] Lustre: 358047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11128.442770] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11135.138763] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11137.310695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11137.338116] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11137.564530] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11145.478584] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11152.479901] Lustre: lustre-OST0001: Not available for connect from 192.168.201.15@tcp (stopping) [11152.871817] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11152.873765] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.902354] Lustre: Skipped 1 previous similar message [11155.960770] Lustre: server umount lustre-OST0001 complete [11157.563240] LustreError: 357147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11157.573746] LustreError: 357147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11163.117625] LustreError: 357145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.134281] LustreError: 357145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11173.353626] LustreError: 358860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11173.370583] LustreError: 358860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11174.886658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11174.900398] LustreError: Skipped 1 previous similar message [11174.907624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11174.958569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11174.961169] Lustre: Skipped 2 previous similar messages [11188.191150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11188.410708] Lustre: server umount lustre-OST0000 complete [11193.828880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11193.838883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11193.841678] Lustre: Skipped 2 previous similar messages [11193.863263] Lustre: Skipped 5 previous similar messages [11199.667039] Lustre: server umount lustre-MDT0000 complete [11203.634124] LustreError: 355531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237018 with bad export cookie 13828966451976872363 [11203.636156] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11203.642456] LustreError: 355531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11210.166862] Lustre: server umount lustre-MDT0001 complete [11221.076276] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11224.113679] Key type lgssc unregistered [11224.472803] LNet: 359909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11224.486035] LNetError: 359909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11224.522161] LNet: Removed LNI 192.168.201.115@tcp [11225.403893] Key type .llcrypt unregistered [11225.405629] Key type ._llcrypt unregistered [11280.282230] Key type ._llcrypt registered [11280.284017] Key type .llcrypt registered [11280.410735] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [11282.023554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11282.074570] alg: No test for adler32 (adler32-zlib) [11283.240722] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [11283.493469] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [11285.256484] Key type lgssc registered [11286.571487] Lustre: Echo OBD driver; http://www.lustre.org/ [11295.295792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11295.328789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11296.784228] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11296.826795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11297.064494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11302.895446] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11312.476861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11312.562448] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11312.589996] Lustre: 362691:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11312.878853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11317.241874] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11324.455602] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11329.143319] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11332.585569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11332.603736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11337.701309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11337.720491] Lustre: Skipped 2 previous similar messages [11337.977276] Lustre: server umount lustre-MDT0000 complete [11342.326296] LustreError: 362685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237157 with bad export cookie 1620471903842294724 [11342.336946] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11342.347356] LustreError: 362685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11342.984375] Lustre: server umount lustre-MDT0001 complete [11353.762910] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:06:07 (1781237167) [11360.918598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11361.505930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11366.888548] LustreError: 364980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.922459] LustreError: 364980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11367.141417] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11371.999888] LustreError: 364981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11377.120043] LustreError: 364980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11378.452924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.704957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11384.020958] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11391.632670] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11397.558083] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11405.151101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11405.475059] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11405.733925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11411.682754] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11417.915884] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11425.724252] Lustre: Permanently deactivating lustre-OST0000 [11425.761880] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11436.112084] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11437.621368] LustreError: 367676:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11437.627732] LustreError: 367676:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11437.631522] LustreError: 367676:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11441.110869] LustreError: 367724:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11441.116358] LustreError: 367724:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11449.601738] Lustre: server umount lustre-OST0000 complete [11452.594606] LustreError: 367975:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11452.600605] LustreError: 367975:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11455.457318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11455.461397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11455.480162] Lustre: Skipped 1 previous similar message [11455.486627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11461.089881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11461.098822] Lustre: Skipped 2 previous similar messages [11466.215065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11469.282293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11471.329862] LustreError: 364980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.351663] LustreError: 364980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11471.447884] Lustre: server umount lustre-MDT0000 complete [11475.621886] LustreError: 364962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237290 with bad export cookie 1620471903842295305 [11475.639286] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11485.318829] Lustre: server umount lustre-MDT0001 complete [11494.638902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11500.054710] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11503.009572] LustreError: 369118:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11510.729398] Lustre: 369458:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.115@tcp [11513.260260] Lustre: server umount MGS complete [11528.681379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11529.188702] LustreError: 370183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11529.205173] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11529.216415] Lustre: Skipped 1 previous similar message [11529.277222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11534.554747] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11539.433313] LustreError: 370184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.449624] LustreError: 370184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11543.589767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11543.949653] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11548.663202] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11554.453882] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11558.144858] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11560.008770] Lustre: Permanently reactivating lustre-OST0000 [11560.025281] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11560.028043] Lustre: Skipped 2 previous similar messages [11589.599186] LustreError: 371785:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11598.544337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11598.983463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11598.988886] Lustre: Skipped 1 previous similar message [11605.517587] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11610.537605] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11619.808939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11619.811521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11619.827127] Lustre: Skipped 1 previous similar message [11619.841654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11619.853741] Lustre: Skipped 2 previous similar messages [11621.193481] Lustre: server umount lustre-OST0000 complete [11624.928274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11624.948113] Lustre: Skipped 2 previous similar messages [11631.079169] Lustre: server umount lustre-MDT0000 complete [11635.176900] LustreError: 371155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.206254] LustreError: 371155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11635.769656] LustreError: 370164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237450 with bad export cookie 1620471903842296355 [11635.775489] LustreError: 370164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11635.779951] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11636.102521] Lustre: server umount lustre-MDT0001 complete [11647.698980] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11650.319851] Key type lgssc unregistered [11650.676980] LNet: 374287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11650.699905] LNetError: 374287:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11651.749355] LNet: Removed LNI 192.168.201.115@tcp [11652.712906] Key type .llcrypt unregistered [11652.714858] Key type ._llcrypt unregistered [11669.493523] Key type ._llcrypt registered [11669.495043] Key type .llcrypt registered [11669.604273] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [11684.912730] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [11686.267534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11686.328234] alg: No test for adler32 (adler32-zlib) [11687.460625] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [11687.698552] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [11689.391201] Key type lgssc registered [11690.727794] Lustre: Echo OBD driver; http://www.lustre.org/ [11728.122955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11728.173515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11729.633629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11729.661513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11729.756868] Lustre: lustre-MDT0000: new disk, initializing [11729.867571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11729.887495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11735.204440] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11747.323584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11747.438527] Lustre: 377409:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11747.509084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11747.518825] Lustre: Skipped 1 previous similar message [11747.613542] Lustre: lustre-MDT0001: new disk, initializing [11747.716174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11747.767302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11747.781618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11752.936640] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11760.551399] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11764.491850] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11771.858683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11772.189686] Lustre: lustre-OST0000: new disk, initializing [11772.212246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11772.231790] Lustre: 379152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11772.371729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11777.654564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11777.675480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11777.787403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11778.746672] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11786.549823] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11791.941532] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11792.304674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11796.723709] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11796.921357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11803.619314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11803.620668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.636749] LustreError: Skipped 1 previous similar message [11803.637423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11803.668419] Lustre: Skipped 1 previous similar message [11808.739940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11808.749465] Lustre: Skipped 1 previous similar message [11812.831857] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11813.103980] Lustre: server umount lustre-OST0000 complete [11818.976192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11818.983863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11818.997887] Lustre: Skipped 1 previous similar message [11819.027074] Lustre: Skipped 2 previous similar messages [11823.546802] Lustre: server umount lustre-MDT0000 complete [11827.783548] LustreError: 378198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237643 with bad export cookie 14812858191477429468 [11827.784882] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11827.802800] LustreError: 378198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11828.489211] Lustre: server umount lustre-MDT0001 complete [11837.522587] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11839.448758] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:14:13 (1781237653) [11846.932273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11847.229634] LustreError: 381465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11847.253185] LustreError: 381465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11847.340504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11852.671764] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11852.774609] LustreError: 381466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11856.864926] LustreError: 381465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11862.006596] LustreError: 381466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.429818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11864.992551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11870.237115] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11875.840831] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11879.779547] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11886.592825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11886.909403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11894.165145] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11901.591764] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11903.504897] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11906.379277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11915.746138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11915.758896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11915.789896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11917.802266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11917.806424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11919.078571] Lustre: server umount lustre-OST0000 complete [11926.496171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11926.512827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11926.544951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11926.551899] Lustre: Skipped 2 previous similar messages [11930.475700] Lustre: server umount lustre-MDT0000 complete [11933.169224] LustreError: 381465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11933.183032] LustreError: 381465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11934.779614] LustreError: 381447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237750 with bad export cookie 14812858191477430350 [11934.787655] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11934.789218] LustreError: 381447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11935.158718] Lustre: server umount lustre-MDT0001 complete [11947.971319] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11951.248514] Key type lgssc unregistered [11951.619971] LNet: 384837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11951.633555] LNetError: 384837:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11951.671316] LNet: Removed LNI 192.168.201.115@tcp [11952.535538] Key type .llcrypt unregistered [11952.540775] Key type ._llcrypt unregistered [11979.530496] Key type ._llcrypt registered [11979.532799] Key type .llcrypt registered [11979.652497] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11980.106436] Key type .llcrypt unregistered [11980.111824] Key type ._llcrypt unregistered [12000.324854] Key type ._llcrypt registered [12000.326850] Key type .llcrypt registered [12000.456495] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12002.001204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12002.105745] alg: No test for adler32 (adler32-zlib) [12003.425699] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [12003.634558] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12005.295522] Key type lgssc registered [12006.543761] Lustre: Echo OBD driver; http://www.lustre.org/ [12014.790422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12014.816398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12016.435256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12021.991863] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12025.827591] LustreError: 387018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12025.859371] LustreError: 387018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12030.955307] LustreError: 387017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12032.245359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12032.507520] LustreError: 387018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12032.551416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12037.600938] LustreError: 387017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12037.617752] LustreError: 387017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12040.108984] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12041.699116] LustreError: 387018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12041.713079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12041.722047] LustreError: 387018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12046.817801] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12051.942200] LustreError: 387017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12051.988041] LustreError: 387017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12054.285204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12054.782559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12059.498571] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12070.375974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12070.389169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12070.391909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12070.398442] Lustre: Skipped 1 previous similar message [12070.414790] Lustre: Skipped 1 previous similar message [12071.353786] Lustre: server umount lustre-OST0000 complete [12075.493835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12075.513520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12079.836429] Lustre: server umount lustre-MDT0000 complete [12080.609520] LustreError: 388886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12080.638640] LustreError: 388886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12083.912241] LustreError: 386997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237899 with bad export cookie 6573539847026759604 [12083.923244] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12084.319331] Lustre: server umount lustre-MDT0001 complete [12097.427845] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12100.734164] Key type lgssc unregistered [12100.979878] LNet: 389999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12100.985793] LNetError: 389999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12102.055067] LNet: Removed LNI 192.168.201.115@tcp [12102.993099] Key type .llcrypt unregistered [12102.995666] Key type ._llcrypt unregistered [12128.758861] Key type ._llcrypt registered [12128.763640] Key type .llcrypt registered [12128.918484] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12130.012128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12130.190462] alg: No test for adler32 (adler32-zlib) [12131.281671] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [12131.509591] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12133.151219] Key type lgssc registered [12134.605510] Lustre: Echo OBD driver; http://www.lustre.org/ [12142.980589] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12143.000685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12158.689100] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12163.807811] LustreError: 391386:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [12190.495224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12198.424396] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12204.491799] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12212.524191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12213.208523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12213.311208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12218.007606] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12218.350705] LustreError: 391411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12218.365060] LustreError: 391411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12223.460457] LustreError: 391412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12226.541827] LustreError: 392312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12226.577784] LustreError: 392312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12228.232142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12228.755483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12234.188559] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12241.833263] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12246.296091] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12254.179041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12254.181498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12254.184646] LustreError: Skipped 1 previous similar message [12254.187049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12254.198533] Lustre: Skipped 1 previous similar message [12255.951970] Lustre: server umount lustre-OST0000 complete [12261.867867] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12261.879882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12261.895175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12261.902431] Lustre: Skipped 1 previous similar message [12262.563966] Lustre: server umount lustre-MDT0000 complete [12264.933082] LustreError: 392315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12264.942855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.500262] LustreError: 391394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238082 with bad export cookie 8258148910242560396 [12267.513665] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12267.525988] LustreError: 391394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12267.949073] Lustre: server umount lustre-MDT0001 complete [12280.169701] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12283.607857] Key type lgssc unregistered [12283.911514] LNet: 394600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12283.919711] LNetError: 394600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12283.935992] LNet: Removed LNI 192.168.201.115@tcp [12284.643518] Key type .llcrypt unregistered [12284.645807] Key type ._llcrypt unregistered [12311.228312] Key type ._llcrypt registered [12311.235778] Key type .llcrypt registered [12311.403757] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12312.690849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12312.765313] alg: No test for adler32 (adler32-zlib) [12313.975407] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [12314.226375] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12315.927697] Key type lgssc registered [12317.252795] Lustre: Echo OBD driver; http://www.lustre.org/ [12325.987509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12326.008930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12327.748688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12332.251648] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12337.148992] LustreError: 396002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12337.170074] LustreError: 396002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12341.870349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12342.248488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12342.498759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12347.787949] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12354.827359] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12359.357964] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12367.731680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12368.254346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12372.343330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12373.729938] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12381.049955] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12387.429333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12387.436046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12387.444622] Lustre: Skipped 2 previous similar messages [12387.461779] Lustre: Skipped 1 previous similar message [12390.679439] Lustre: server umount lustre-MDT0000 complete [12392.884526] LustreError: 398240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.15@tcp arrived at 1781238208 with bad export cookie 8295690856618039866 [12392.910756] LustreError: 398240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12392.939437] LustreError: 395997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12392.959387] LustreError: 395997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12395.487755] LustreError: 395983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238210 with bad export cookie 8295690856618038977 [12395.515829] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12398.053092] LustreError: 395998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12398.061144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12398.100163] LustreError: 395998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12398.138191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12398.140957] Lustre: Skipped 1 previous similar message [12400.103171] LustreError: 396001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.137302] LustreError: 396001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12402.092684] Lustre: server umount lustre-OST0000 complete [12405.223061] LustreError: 396955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12405.248240] LustreError: 396955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12412.127454] LustreError: 396714:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003307eb75 x1867763552386688/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12412.563069] Lustre: server umount lustre-MDT0001 complete [12425.535191] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12429.259890] Key type lgssc unregistered [12429.658332] LNet: 399329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12429.678062] LNetError: 399329:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12429.715668] LNet: Removed LNI 192.168.201.115@tcp [12430.741566] Key type .llcrypt unregistered [12430.747868] Key type ._llcrypt unregistered [12455.513959] Key type ._llcrypt registered [12455.522145] Key type .llcrypt registered [12455.664411] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12457.119637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12457.160017] alg: No test for adler32 (adler32-zlib) [12458.370203] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [12458.661091] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12460.354235] Key type lgssc registered [12461.825496] Lustre: Echo OBD driver; http://www.lustre.org/ [12470.647779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12470.663218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12472.204808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12477.164771] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12482.529773] LustreError: 400734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12482.563343] LustreError: 400734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12486.285190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12486.692911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12491.511752] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12497.929538] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12501.853772] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12508.694693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12509.222589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12516.105317] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12517.427893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12522.294774] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12527.586260] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12527.592107] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12527.601451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12527.616761] Lustre: Skipped 1 previous similar message [12530.245185] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [12532.705734] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12532.712768] Lustre: Skipped 1 previous similar message [12533.440308] Lustre: server umount lustre-MDT0001 complete [12535.344385] LustreError: 400731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12537.826654] LustreError: 401446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12537.847643] LustreError: 401446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12542.948930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12542.954266] LustreError: 400734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12542.958855] Lustre: Skipped 1 previous similar message [12542.963877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12545.682961] Lustre: server umount lustre-OST0000 complete [12548.067062] LustreError: 401446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12549.943858] Lustre: server umount lustre-MDT0000 complete [12565.290977] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12568.939286] Key type lgssc unregistered [12569.361745] LNet: 404064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12569.369461] LNetError: 404064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12569.403114] LNet: Removed LNI 192.168.201.115@tcp [12570.429272] Key type .llcrypt unregistered [12570.430442] Key type ._llcrypt unregistered [12599.040967] Key type ._llcrypt registered [12599.049217] Key type .llcrypt registered [12599.212934] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [12616.855549] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12618.021557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12618.128563] alg: No test for adler32 (adler32-zlib) [12619.396382] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [12619.650850] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12621.287175] Key type lgssc registered [12622.697759] Lustre: Echo OBD driver; http://www.lustre.org/ [12661.315779] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12677.322466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12677.348291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12678.576860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12678.606573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12678.669323] Lustre: lustre-MDT0000: new disk, initializing [12678.746470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12678.764793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12683.700156] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12701.173030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12701.378109] Lustre: lustre-OST0000: new disk, initializing [12701.381652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12701.385441] Lustre: Skipped 1 previous similar message [12701.389392] Lustre: 409033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12701.460163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12707.801820] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12711.076723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12711.085222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12711.111755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12716.338068] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12733.453124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12733.612962] Lustre: 408032:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12733.721749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12733.823895] Lustre: lustre-MDT0001: new disk, initializing [12733.983482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12734.049528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12734.074531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12738.462321] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12748.171663] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12748.452432] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12752.789753] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12752.976288] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12758.021100] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12758.035517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12760.672558] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [12763.116203] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12763.116967] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12763.134188] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12764.586508] Lustre: server umount lustre-MDT0001 complete [12773.359538] LustreError: 408045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12773.379453] LustreError: 408045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12778.481665] LustreError: 408039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12778.507749] LustreError: 408039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12783.072697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12783.081528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12783.105963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12783.124606] Lustre: Skipped 1 previous similar message [12783.586659] LustreError: 408044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12788.593120] Lustre: server umount lustre-OST0000 complete [12788.710863] LustreError: 408045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.630660] Lustre: server umount lustre-MDT0000 complete [12813.460477] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12815.943583] Key type lgssc unregistered [12816.257600] LNet: 412597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12816.262634] LNetError: 412597:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12816.275631] LNet: Removed LNI 192.168.201.115@tcp [12817.172281] Key type .llcrypt unregistered [12817.175862] Key type ._llcrypt unregistered [12855.286745] Key type ._llcrypt registered [12855.288490] Key type .llcrypt registered [12855.384487] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12855.685415] Key type .llcrypt unregistered [12855.688909] Key type ._llcrypt unregistered [12872.633654] Key type ._llcrypt registered [12872.638372] Key type .llcrypt registered [12872.799977] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12873.997152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12874.251471] alg: No test for adler32 (adler32-zlib) [12875.223629] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [12875.447256] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12877.111478] Key type lgssc registered [12878.602866] Lustre: Echo OBD driver; http://www.lustre.org/ [12909.082216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12909.157801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12910.537719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12910.577082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12910.704652] Lustre: lustre-MDT0000: new disk, initializing [12910.833381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12910.876823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12915.970507] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12926.737449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12926.871390] Lustre: 415960:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12926.928318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12926.930745] Lustre: Skipped 1 previous similar message [12927.043345] Lustre: lustre-MDT0001: new disk, initializing [12927.209526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12927.286431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12927.305517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12933.280413] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12941.509749] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12946.112323] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12954.073312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12954.356387] Lustre: lustre-OST0000: new disk, initializing [12954.359970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12954.365635] Lustre: 417704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12954.505475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12961.675302] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12963.923194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12963.946168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12964.030187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12970.148132] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12978.658434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12978.675227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12978.697711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12979.680337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12979.692958] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12979.702219] Lustre: Skipped 1 previous similar message [12982.221045] Lustre: server umount lustre-MDT0000 complete [12986.335799] LustreError: 415952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238801 with bad export cookie 4977006747749027301 [12986.338131] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12986.378816] LustreError: 415952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12986.788569] Lustre: server umount lustre-MDT0001 complete [13000.235444] Lustre: server umount lustre-OST0000 complete [13010.270442] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:33:44 (1781238824) [13017.519152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13017.792269] LustreError: 419619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13017.806547] LustreError: 419619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13017.850924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13021.239408] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13023.217900] LustreError: 419620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13028.321640] LustreError: 419619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13029.776724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13030.205743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13034.517022] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13040.952784] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13045.631067] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13066.100337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13066.403983] Lustre: Found index 0 for lustre-OST0000, updating log [13066.766736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13074.829478] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13081.025589] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13089.333540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13099.493513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13099.512816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13099.535170] Lustre: Skipped 1 previous similar message [13099.536789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13099.545071] Lustre: Skipped 3 previous similar messages [13106.661660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13106.671225] Lustre: Skipped 2 previous similar messages [13112.291267] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13112.557133] Lustre: server umount lustre-OST0000 complete [13125.345378] LNet: 1 local NIs in recovery (showing 1): 192.168.201.115@tcp [13132.768994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13132.773837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13132.797750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13134.572813] Lustre: server umount lustre-MDT0000 complete [13138.403827] LustreError: 419620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13138.427503] LustreError: 419620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13138.497961] LustreError: 419600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238953 with bad export cookie 4977006747749028694 [13138.522101] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13139.084545] Lustre: server umount lustre-MDT0001 complete [13148.119804] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:36:02 (1781238962) [13189.855207] LNet: 1 local NIs in recovery (showing 1): 192.168.201.115@tcp [13771.282154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13771.851444] LustreError: 425713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13771.960461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13777.377356] LustreError: 425714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13777.444446] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13782.502864] LustreError: 425713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13787.772977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13788.192837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13792.848955] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13799.764384] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13803.519935] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13823.857865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13824.139824] Lustre: Found index 0 for lustre-OST0000, updating log [13824.441911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13829.855238] LNet: 1 local NIs in recovery (showing 1): 192.168.201.115@tcp [13830.510756] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13836.697239] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13838.952762] Lustre: server umount lustre-OST0000 complete [13844.448224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13844.467486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13844.476558] Lustre: Skipped 1 previous similar message [13844.485586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13844.492971] Lustre: Skipped 2 previous similar messages [13845.542317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13845.553882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13847.653048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13847.659957] Lustre: Skipped 4 previous similar messages [13851.816671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13851.819043] Lustre: Skipped 6 previous similar messages [13858.271389] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13858.793085] Lustre: server umount lustre-MDT0000 complete [13858.990913] LustreError: 425708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13859.021836] LustreError: 425708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13863.139028] LustreError: 425713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13863.169865] LustreError: 425713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [13864.890195] LustreError: 425694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239680 with bad export cookie 4977006747749029779 [13864.893509] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13864.900708] LustreError: 425694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [13865.234972] Lustre: server umount lustre-MDT0001 complete [13893.343292] LNet: 1 local NIs in recovery (showing 1): 192.168.201.115@tcp [13898.511996] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [13909.569669] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [13949.479386] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:49:23 (1781239763) [13968.398072] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13971.497717] Key type lgssc unregistered [13971.766605] LNet: 431931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13971.772896] LNetError: 431931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14087.533902] LNet: Removed LNI 192.168.201.115@tcp [14088.958932] Key type .llcrypt unregistered [14088.966161] Key type ._llcrypt unregistered [14107.182780] Key type ._llcrypt registered [14107.184808] Key type .llcrypt registered [14107.283707] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [14107.934916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14108.038439] alg: No test for adler32 (adler32-zlib) [14109.241523] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [14109.585276] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [14111.383248] Key type lgssc registered [14112.790564] Lustre: Echo OBD driver; http://www.lustre.org/ [14117.329547] LNet: Added LNI 192.168.201.115@tcp100 [8/256/0/180] [14118.445436] LNet: Added LNI 192.168.201.115@tcp101 [8/256/0/180] [14119.337689] LNet: Added LNI 192.168.201.115@tcp102 [8/256/0/180] [14120.176113] LNet: Added LNI 192.168.201.115@tcp103 [8/256/0/180] [14121.356353] LNet: Added LNI 192.168.201.115@tcp104 [8/256/0/180] [14122.234544] LNet: Added LNI 192.168.201.115@tcp105 [8/256/0/180] [14123.296482] LNet: Added LNI 192.168.201.115@tcp106 [8/256/0/180] [14124.272867] LNet: Added LNI 192.168.201.115@tcp107 [8/256/0/180] [14125.130366] LNet: Added LNI 192.168.201.115@tcp108 [8/256/0/180] [14125.915483] LNet: Added LNI 192.168.201.115@tcp109 [8/256/0/180] [14126.649956] LNet: Added LNI 192.168.201.115@tcp110 [8/256/0/180] [14127.551453] LNet: Added LNI 192.168.201.115@tcp111 [8/256/0/180] [14128.381263] LNet: Added LNI 192.168.201.115@tcp112 [8/256/0/180] [14129.289746] LNet: Added LNI 192.168.201.115@tcp113 [8/256/0/180] [14129.908877] LNet: Added LNI 192.168.201.115@tcp114 [8/256/0/180] [14130.771139] LNet: Added LNI 192.168.201.115@tcp115 [8/256/0/180] [14131.649755] LNet: Added LNI 192.168.201.115@tcp116 [8/256/0/180] [14132.528218] LNet: Added LNI 192.168.201.115@tcp117 [8/256/0/180] [14133.417925] LNet: Added LNI 192.168.201.115@tcp118 [8/256/0/180] [14134.350651] LNet: Added LNI 192.168.201.115@tcp119 [8/256/0/180] [14135.170241] LNet: Added LNI 192.168.201.115@tcp120 [8/256/0/180] [14135.788172] LNet: Added LNI 192.168.201.115@tcp121 [8/256/0/180] [14136.436804] LNet: Added LNI 192.168.201.115@tcp122 [8/256/0/180] [14137.431175] LNet: Added LNI 192.168.201.115@tcp123 [8/256/0/180] [14138.216189] LNet: Added LNI 192.168.201.115@tcp124 [8/256/0/180] [14139.193621] LNet: Added LNI 192.168.201.115@tcp125 [8/256/0/180] [14140.157970] LNet: Added LNI 192.168.201.115@tcp126 [8/256/0/180] [14141.205552] LNet: Added LNI 192.168.201.115@tcp127 [8/256/0/180] [14142.207835] LNet: Added LNI 192.168.201.115@tcp128 [8/256/0/180] [14143.213311] LNet: Added LNI 192.168.201.115@tcp129 [8/256/0/180] [14144.131585] LNet: Added LNI 192.168.201.115@tcp130 [8/256/0/180] [14144.740550] LNet: Added LNI 192.168.201.115@tcp131 [8/256/0/180] [14145.578968] LNet: Added LNI 192.168.201.115@tcp132 [8/256/0/180] [14146.511093] LNet: Added LNI 192.168.201.115@tcp133 [8/256/0/180] [14147.385523] LNet: Added LNI 192.168.201.115@tcp134 [8/256/0/180] [14148.230344] LNet: Added LNI 192.168.201.115@tcp135 [8/256/0/180] [14155.249529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14155.270276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14156.454424] Lustre: 434947:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14156.554567] LNetError: 434968:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14156.565477] LNetError: Protocol error connecting to 192.168.201.115@tcp118 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14156.584359] LNetError: Skipped 7 previous similar messages [14156.705826] LNet: 6 local NIs in recovery (showing 6): 192.168.201.115@tcp122, 192.168.201.115@tcp123, 192.168.201.115@tcp125, 192.168.201.115@tcp126, 192.168.201.115@tcp124, 192.168.201.115@tcp129 [14156.766392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14156.807590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14156.888601] Lustre: lustre-MDT0000: new disk, initializing [14157.045424] Lustre: 435004:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14157.052283] Lustre: 435004:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14157.086591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14157.114872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14157.748190] LNetError: 432408:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14157.763315] LNetError: 432408:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [14157.772943] LNetError: Protocol error connecting to 192.168.201.115@tcp122 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14157.797785] LNetError: Skipped 7 previous similar messages [14159.791163] LNetError: 434975:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14159.802842] LNetError: 434975:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [14159.822701] LNetError: Protocol error connecting to 192.168.201.115@tcp122 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14162.126691] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14163.875475] LNetError: 434967:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14163.889256] LNetError: Protocol error connecting to 192.168.201.115@tcp122 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14171.040644] LNetError: 434973:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14171.058181] LNetError: Protocol error connecting to 192.168.201.115@tcp122 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14173.882093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14174.112389] Lustre: 434963:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14174.194463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14174.196788] Lustre: Skipped 1 previous similar message [14174.279220] Lustre: lustre-MDT0001: new disk, initializing [14174.317807] Lustre: 435750:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14174.327756] Lustre: 435750:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14174.437078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14174.476286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14174.489913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14179.023459] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14183.985715] Lustre: 436192:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14183.999794] Lustre: 436192:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14185.695456] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14186.402388] LNetError: 434976:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14186.407237] LNetError: Protocol error connecting to 192.168.201.115@tcp120 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14189.231440] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14195.725935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14196.084983] Lustre: lustre-OST0000: new disk, initializing [14196.088335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14196.095215] Lustre: 436714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14196.204381] Lustre: 436719:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14196.209966] Lustre: 436719:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14196.251689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14202.795218] LNetError: 432406:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14202.811561] LNetError: 432406:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [14202.820462] LNetError: Protocol error connecting to 192.168.201.115@tcp126 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14202.827522] LNetError: Skipped 14 previous similar messages [14202.947908] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14205.979382] Lustre: 437160:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14205.988338] Lustre: 437160:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14206.081598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14206.097515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14206.213845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14210.305547] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14216.097152] LNet: 1 local NIs in recovery (showing 1): 192.168.201.115@tcp122 [14221.798115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14221.801441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14221.803167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14221.805632] Lustre: Skipped 1 previous similar message [14221.834899] LustreError: Skipped 1 previous similar message [14223.749332] Lustre: server umount lustre-OST0000 complete [14231.008849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14231.022357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14231.047426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14231.052139] Lustre: Skipped 1 previous similar message [14232.038481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14232.050436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14232.056373] Lustre: Skipped 1 previous similar message [14233.806271] Lustre: server umount lustre-MDT0000 complete [14235.566902] LNetError: 434973:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14235.588153] LNetError: 434973:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 25 previous similar messages [14235.602686] LNetError: Protocol error connecting to 192.168.201.115@tcp117 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14235.608093] LNetError: Skipped 25 previous similar messages [14237.874933] LustreError: 434956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240053 with bad export cookie 5983551694565603891 [14237.882905] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14238.553034] Lustre: server umount lustre-MDT0001 complete [14253.737616] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [14257.245523] Key type lgssc unregistered [14257.672477] LNet: 438497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14257.685219] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14257.701745] LNet: Removed LNI 192.168.201.115@tcp [14258.719260] LNet: Removed LNI 192.168.201.115@tcp100 [14258.721186] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14258.746322] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14258.755625] LNet: Removed LNI 192.168.201.115@tcp101 [14259.807983] LNet: Removed LNI 192.168.201.115@tcp102 [14259.817833] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14259.829449] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14259.838200] LNet: Removed LNI 192.168.201.115@tcp103 [14260.896065] LNet: Removed LNI 192.168.201.115@tcp104 [14261.919942] LNet: Removed LNI 192.168.201.115@tcp105 [14261.922242] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14261.931813] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14261.949466] LNet: Removed LNI 192.168.201.115@tcp106 [14263.008668] LNet: Removed LNI 192.168.201.115@tcp107 [14264.031164] LNet: Removed LNI 192.168.201.115@tcp108 [14265.119368] LNet: Removed LNI 192.168.201.115@tcp109 [14266.143694] LNet: Removed LNI 192.168.201.115@tcp110 [14266.150145] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14266.160121] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14266.184844] LNet: Removed LNI 192.168.201.115@tcp111 [14267.235238] LNet: Removed LNI 192.168.201.115@tcp112 [14268.255179] LNet: Removed LNI 192.168.201.115@tcp113 [14269.279170] LNet: Removed LNI 192.168.201.115@tcp114 [14270.303157] LNet: Removed LNI 192.168.201.115@tcp115 [14271.327321] LNet: Removed LNI 192.168.201.115@tcp116 [14272.351175] LNet: Removed LNI 192.168.201.115@tcp117 [14273.378042] LNet: Removed LNI 192.168.201.115@tcp118 [14274.399179] LNet: Removed LNI 192.168.201.115@tcp119 [14274.405790] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14274.426653] LNetError: 438497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14274.440713] LNet: Removed LNI 192.168.201.115@tcp120 [14275.487471] LNet: Removed LNI 192.168.201.115@tcp121 [14275.489459] LNet: Removed LNI 192.168.201.115@tcp122 [14276.511124] LNet: Removed LNI 192.168.201.115@tcp123 [14277.535822] LNet: Removed LNI 192.168.201.115@tcp124 [14278.559449] LNet: Removed LNI 192.168.201.115@tcp125 [14279.583553] LNet: Removed LNI 192.168.201.115@tcp126 [14280.607168] LNet: Removed LNI 192.168.201.115@tcp127 [14281.635933] LNet: Removed LNI 192.168.201.115@tcp128 [14282.655398] LNet: Removed LNI 192.168.201.115@tcp129 [14282.661115] LNet: Removed LNI 192.168.201.115@tcp130 [14282.666660] LNet: Removed LNI 192.168.201.115@tcp131 [14282.672103] LNet: Removed LNI 192.168.201.115@tcp132 [14282.677905] LNet: Removed LNI 192.168.201.115@tcp133 [14282.686838] LNet: Removed LNI 192.168.201.115@tcp134 [14282.701363] LNet: Removed LNI 192.168.201.115@tcp135 [14283.514558] Key type .llcrypt unregistered [14283.516148] Key type ._llcrypt unregistered [14306.686658] Key type ._llcrypt registered [14306.688050] Key type .llcrypt registered [14306.793876] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [14320.955398] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [14322.000514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14322.269806] alg: No test for adler32 (adler32-zlib) [14323.257158] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [14323.468349] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [14325.111279] Key type lgssc registered [14326.225707] Lustre: Echo OBD driver; http://www.lustre.org/ [14358.051797] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:56:12 (1781240172) [14378.359221] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [14381.873759] Key type lgssc unregistered [14382.237555] LNet: 442534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14382.242764] LNetError: 442534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14382.254382] LNet: Removed LNI 192.168.201.115@tcp [14383.182381] Key type .llcrypt unregistered [14383.186149] Key type ._llcrypt unregistered [14400.083419] Key type ._llcrypt registered [14400.087482] Key type .llcrypt registered [14400.200814] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [14401.477195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14401.511452] alg: No test for adler32 (adler32-zlib) [14402.753278] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [14403.014590] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [14404.696630] Key type lgssc registered [14406.183024] Lustre: Echo OBD driver; http://www.lustre.org/ [14416.654180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14416.705699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14418.086369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14418.149543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14418.308608] Lustre: lustre-MDT0000: new disk, initializing [14418.461095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14418.502219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14423.387788] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14434.718542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14434.781805] Lustre: 443730:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14434.820612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14434.823646] Lustre: Skipped 1 previous similar message [14434.874672] Lustre: lustre-MDT0001: new disk, initializing [14434.967874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14435.005719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14435.016159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14440.785545] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14449.100901] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14452.570478] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14459.121724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14459.516523] Lustre: lustre-OST0000: new disk, initializing [14459.522607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14459.531682] Lustre: 445474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14459.658432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14466.689974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14466.698551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14466.703780] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14466.821527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14475.377456] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14479.910593] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing lnet_if_list [14486.227085] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14486.324986] LNet: Added LNI 192.168.201.115@tcp10 [8/256/0/180] [14493.542341] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14493.597628] LNet: Added LNI 192.168.201.115@tcp11 [8/256/0/180] [14499.090816] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14499.115242] LNet: Added LNI 192.168.201.115@tcp12 [8/256/0/180] [14503.595217] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14503.610594] LNet: Added LNI 192.168.201.115@tcp13 [8/256/0/180] [14508.102151] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14508.137552] LNet: Added LNI 192.168.201.115@tcp14 [8/256/0/180] [14511.769231] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14511.790961] LNet: Added LNI 192.168.201.115@tcp15 [8/256/0/180] [14515.819582] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14515.847804] LNet: Added LNI 192.168.201.115@tcp16 [8/256/0/180] [14520.039512] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14520.065459] LNet: Added LNI 192.168.201.115@tcp17 [8/256/0/180] [14524.051752] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14524.076865] LNet: Added LNI 192.168.201.115@tcp18 [8/256/0/180] [14528.196979] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14528.226982] LNet: Added LNI 192.168.201.115@tcp19 [8/256/0/180] [14532.578930] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14532.614144] LNet: Added LNI 192.168.201.115@tcp20 [8/256/0/180] [14537.396800] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14537.409285] LNet: Added LNI 192.168.201.115@tcp21 [8/256/0/180] [14542.469675] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14542.499521] LNet: Added LNI 192.168.201.115@tcp22 [8/256/0/180] [14547.680851] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14547.711332] LNet: Added LNI 192.168.201.115@tcp23 [8/256/0/180] [14552.514742] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14552.534815] LNet: Added LNI 192.168.201.115@tcp24 [8/256/0/180] [14558.061755] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14558.086612] LNet: Added LNI 192.168.201.115@tcp25 [8/256/0/180] [14563.615310] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14563.649537] LNet: Added LNI 192.168.201.115@tcp26 [8/256/0/180] [14569.103940] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14569.149321] LNet: Added LNI 192.168.201.115@tcp27 [8/256/0/180] [14576.334530] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14576.387427] LNet: Added LNI 192.168.201.115@tcp28 [8/256/0/180] [14583.303286] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14583.391200] LNet: Added LNI 192.168.201.115@tcp29 [8/256/0/180] [14589.997256] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14590.018201] LNet: Added LNI 192.168.201.115@tcp30 [8/256/0/180] [14595.900563] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14595.936735] LNet: Added LNI 192.168.201.115@tcp31 [8/256/0/180] [14601.727806] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14601.765909] LNet: Added LNI 192.168.201.115@tcp32 [8/256/0/180] [14608.438371] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14608.487064] LNet: Added LNI 192.168.201.115@tcp33 [8/256/0/180] [14615.075740] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14615.131294] LNet: Added LNI 192.168.201.115@tcp34 [8/256/0/180] [14622.583611] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14622.604962] LNet: Added LNI 192.168.201.115@tcp35 [8/256/0/180] [14630.246330] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14630.274381] LNet: Added LNI 192.168.201.115@tcp36 [8/256/0/180] [14637.430746] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14637.470591] LNet: Added LNI 192.168.201.115@tcp37 [8/256/0/180] [14644.341738] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14644.362416] LNet: Added LNI 192.168.201.115@tcp38 [8/256/0/180] [14652.222137] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14652.243706] LNet: Added LNI 192.168.201.115@tcp39 [8/256/0/180] [14659.680327] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14659.704748] LNet: Added LNI 192.168.201.115@tcp40 [8/256/0/180] [14668.188599] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14668.212837] LNet: Added LNI 192.168.201.115@tcp41 [8/256/0/180] [14676.809833] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14676.845439] LNet: Added LNI 192.168.201.115@tcp42 [8/256/0/180] [14684.865577] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14684.908798] LNet: Added LNI 192.168.201.115@tcp43 [8/256/0/180] [14693.532595] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14693.568583] LNet: Added LNI 192.168.201.115@tcp44 [8/256/0/180] [14701.877762] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14701.924940] LNet: Added LNI 192.168.201.115@tcp45 [8/256/0/180] [14708.564747] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14708.580052] LNet: Added LNI 192.168.201.115@tcp46 [8/256/0/180] [14716.225507] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14716.259962] LNet: Added LNI 192.168.201.115@tcp47 [8/256/0/180] [14724.893677] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14724.927576] LNet: Added LNI 192.168.201.115@tcp48 [8/256/0/180] [14732.549607] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14732.584928] LNet: Added LNI 192.168.201.115@tcp49 [8/256/0/180] [14741.487991] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14741.526521] LNet: Added LNI 192.168.201.115@tcp50 [8/256/0/180] [14761.954435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14761.964347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14761.980389] Lustre: Skipped 1 previous similar message [14765.758308] Lustre: server umount lustre-MDT0000 complete [14769.555693] LustreError: 443723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240584 with bad export cookie 17140928654154189660 [14769.558382] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14769.565171] LustreError: 443723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14769.994486] Lustre: server umount lustre-MDT0001 complete [14786.773382] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [14788.577094] Lustre: 443004:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240587/real 1781240587] req@0000000029b9253f x1867765742995200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781240603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14788.628099] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14788.653412] Lustre: Skipped 1 previous similar message [14790.623190] Lustre: 443001:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240589/real 1781240589] req@00000000cf34f9e4 x1867765742995328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781240605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14793.697203] Lustre: 443002:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240592/real 1781240592] req@000000009a6f65a3 x1867765742995456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781240608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14795.871306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14799.903516] Lustre: 443004:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240599/real 1781240599] req@00000000ae4698fb x1867765742995840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781240615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14799.936159] Lustre: 443004:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14814.304582] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede0cfcb5f2cd565 [14814.313635] Lustre: MGC192.168.201.115@tcp: Connection restored to 0@lo (at 0@lo) [14814.593715] LustreError: 445466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14814.629746] LustreError: 445466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14814.677274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14817.705557] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14817.706251] LustreError: 445467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14818.725466] LustreError: 445466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14819.099583] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14823.942060] LustreError: 445467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14827.205630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14827.486577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14831.420652] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14833.641298] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14836.672972] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14840.384899] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14858.212874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14858.222107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14858.238997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14858.244167] Lustre: Skipped 2 previous similar messages [14859.240659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14863.194413] Lustre: server umount lustre-OST0000 complete [14868.456288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14868.462996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14868.476182] Lustre: Skipped 1 previous similar message [14868.483718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14868.489866] Lustre: Skipped 1 previous similar message [14874.594693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14874.602937] Lustre: Skipped 3 previous similar messages [14881.248558] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14881.519450] Lustre: server umount lustre-MDT0000 complete [14883.813916] LustreError: 468886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14883.835566] LustreError: 468886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14885.305406] LustreError: 443722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240700 with bad export cookie 17140928654154192229 [14885.307254] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14885.322437] LustreError: 443722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14885.556041] Lustre: server umount lustre-MDT0001 complete [14894.708359] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [14896.985554] Key type lgssc unregistered [14897.262692] LNet: 471419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14897.269896] LNetError: 471419:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14897.275929] LNet: Removed LNI 192.168.201.115@tcp [14897.277311] LNet: Removed LNI 192.168.201.115@tcp10 [14897.278357] LNet: Removed LNI 192.168.201.115@tcp11 [14897.283240] LNet: Removed LNI 192.168.201.115@tcp12 [14897.284919] LNet: Removed LNI 192.168.201.115@tcp13 [14897.289374] LNet: Removed LNI 192.168.201.115@tcp14 [14897.291369] LNet: Removed LNI 192.168.201.115@tcp15 [14897.292898] LNet: Removed LNI 192.168.201.115@tcp16 [14897.299775] LNet: Removed LNI 192.168.201.115@tcp17 [14897.303315] LNet: Removed LNI 192.168.201.115@tcp18 [14897.307507] LNet: Removed LNI 192.168.201.115@tcp19 [14897.311651] LNet: Removed LNI 192.168.201.115@tcp20 [14897.316613] LNet: Removed LNI 192.168.201.115@tcp21 [14897.321877] LNet: Removed LNI 192.168.201.115@tcp22 [14897.324496] LNet: Removed LNI 192.168.201.115@tcp23 [14897.328247] LNet: Removed LNI 192.168.201.115@tcp24 [14897.330423] LNet: Removed LNI 192.168.201.115@tcp25 [14897.332699] LNet: Removed LNI 192.168.201.115@tcp26 [14897.334715] LNet: Removed LNI 192.168.201.115@tcp27 [14897.336718] LNet: Removed LNI 192.168.201.115@tcp28 [14897.338683] LNet: Removed LNI 192.168.201.115@tcp29 [14897.343433] LNet: Removed LNI 192.168.201.115@tcp30 [14897.345212] LNet: Removed LNI 192.168.201.115@tcp31 [14897.347542] LNet: Removed LNI 192.168.201.115@tcp32 [14897.349893] LNet: Removed LNI 192.168.201.115@tcp33 [14897.351373] LNet: Removed LNI 192.168.201.115@tcp34 [14897.352743] LNet: Removed LNI 192.168.201.115@tcp35 [14897.354338] LNet: Removed LNI 192.168.201.115@tcp36 [14897.355886] LNet: Removed LNI 192.168.201.115@tcp37 [14897.357711] LNet: Removed LNI 192.168.201.115@tcp38 [14897.358889] LNet: Removed LNI 192.168.201.115@tcp39 [14897.359957] LNet: Removed LNI 192.168.201.115@tcp40 [14897.361211] LNet: Removed LNI 192.168.201.115@tcp41 [14897.362306] LNet: Removed LNI 192.168.201.115@tcp42 [14897.363416] LNet: Removed LNI 192.168.201.115@tcp43 [14897.364501] LNet: Removed LNI 192.168.201.115@tcp44 [14897.365701] LNet: Removed LNI 192.168.201.115@tcp45 [14897.367468] LNet: Removed LNI 192.168.201.115@tcp46 [14897.368726] LNet: Removed LNI 192.168.201.115@tcp47 [14897.369741] LNet: Removed LNI 192.168.201.115@tcp48 [14897.370879] LNet: Removed LNI 192.168.201.115@tcp49 [14897.381495] LNet: Removed LNI 192.168.201.115@tcp50 [14897.909634] Key type .llcrypt unregistered [14897.915060] Key type ._llcrypt unregistered [14917.311451] Key type ._llcrypt registered [14917.318679] Key type .llcrypt registered [14917.469039] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [14929.992482] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [14930.713209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14930.868025] alg: No test for adler32 (adler32-zlib) [14932.000243] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [14932.212330] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [14933.887315] Key type lgssc registered [14934.904123] Lustre: Echo OBD driver; http://www.lustre.org/ [14968.847218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14968.889153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14970.197168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14970.225062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14970.286258] Lustre: lustre-MDT0000: new disk, initializing [14970.357304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14970.397643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14974.900871] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14985.966917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14986.049130] Lustre: 474561:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14986.080825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14986.082955] Lustre: Skipped 1 previous similar message [14986.187260] Lustre: lustre-MDT0001: new disk, initializing [14986.287474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14986.324580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14986.333856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14990.335953] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14997.678803] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15001.462097] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15009.354905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15009.526287] Lustre: *** cfs_fail_loc=90f, val=0*** [15009.626633] Lustre: lustre-OST0000: new disk, initializing [15009.629469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15009.637866] Lustre: 476352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15009.728319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15015.160318] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15019.084670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15019.097949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15019.212026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15022.457790] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15029.753310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15029.761509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15029.767967] LustreError: Skipped 1 previous similar message [15029.769133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15029.790624] Lustre: Skipped 1 previous similar message [15034.879462] Lustre: server umount lustre-OST0000 complete [15039.969371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15039.991377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15040.004329] Lustre: Skipped 2 previous similar messages [15045.091929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15045.097989] Lustre: Skipped 1 previous similar message [15050.210076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15050.217859] Lustre: Skipped 1 previous similar message [15053.279129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15053.399737] Lustre: server umount lustre-MDT0000 complete [15056.958749] LustreError: 476351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240872 with bad export cookie 10946266183592202288 [15056.965788] LustreError: 476351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15056.970784] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15057.378791] Lustre: server umount lustre-MDT0001 complete [15067.464814] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [15069.949305] Key type lgssc unregistered [15070.129788] LNet: 478127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15070.135297] LNetError: 478127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15070.148309] LNet: Removed LNI 192.168.201.115@tcp [15070.842794] Key type .llcrypt unregistered [15070.846131] Key type ._llcrypt unregistered [15091.036626] Key type ._llcrypt registered [15091.038722] Key type .llcrypt registered [15091.140355] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [15092.552292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15092.579130] alg: No test for adler32 (adler32-zlib) [15093.876819] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [15094.193908] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [15095.911827] Key type lgssc registered [15097.044732] Lustre: Echo OBD driver; http://www.lustre.org/ [15106.142723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15106.158931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15107.625129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15112.305219] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15117.793575] LustreError: 479382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15117.817660] LustreError: 479382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15121.282699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15121.773929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15126.850078] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15133.379931] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15137.680847] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15146.308754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15146.438201] LustreError: 480977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15146.809537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15153.404468] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15160.599877] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15171.785284] loop0: detected capacity change from 0 to 204800000 [15172.047195] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15172.067282] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15172.083266] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15178.621852] loop0: detected capacity change from 0 to 204800000 [15178.827165] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15178.841748] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15178.859342] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15184.329629] loop0: detected capacity change from 0 to 204800000 [15190.268962] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15190.417981] Lustre: fs15246-OST0000: new disk, initializing [15190.420100] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15190.430241] Lustre: 482636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15190.479565] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15196.315412] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15206.482424] loop1: detected capacity change from 0 to 204800000 [15212.232684] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15212.347334] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15212.397925] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15212.458610] Lustre: fs15246-MDT0000: new disk, initializing [15212.602096] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15212.636946] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15212.784591] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15213.870933] Lustre: 483743:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000bdcb97e1 x1867766467442176/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781241045 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15217.653555] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15222.948699] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15222.960688] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15222.988403] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15237.616705] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15237.632133] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15242.671059] Lustre: server umount fs15246-MDT0000 complete [15242.736735] LustreError: 479377:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15247.870734] LustreError: 483087:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15251.435178] Lustre: server umount fs15246-OST0000 complete [15263.204199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15263.208822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15263.220985] Lustre: Skipped 1 previous similar message [15267.638343] Lustre: server umount lustre-OST0000 complete [15273.443316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15273.452362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15273.459295] Lustre: Skipped 1 previous similar message [15273.469938] Lustre: Skipped 1 previous similar message [15278.564698] LustreError: 479382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15278.584832] LustreError: 479382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15279.507253] Lustre: server umount lustre-MDT0000 complete [15283.231643] LustreError: 486070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241098 with bad export cookie 1326881672211056923 [15283.237702] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15283.250105] LustreError: 486070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15283.735050] Lustre: server umount lustre-MDT0001 complete [15296.477467] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [15300.535405] Key type lgssc unregistered [15301.173759] LNet: 486805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15301.179778] LNetError: 486805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15301.222805] LNet: Removed LNI 192.168.201.115@tcp [15302.303926] Key type .llcrypt unregistered [15302.306510] Key type ._llcrypt unregistered [15324.622896] Key type ._llcrypt registered [15324.625196] Key type .llcrypt registered [15324.730954] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [15326.091120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15326.170069] alg: No test for adler32 (adler32-zlib) [15327.152952] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [15327.330727] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [15328.999189] Key type lgssc registered [15329.834613] Lustre: Echo OBD driver; http://www.lustre.org/ [15370.658418] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [15379.509957] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [15414.097400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15414.125059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15415.450863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15415.497578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15415.571731] Lustre: lustre-MDT0000: new disk, initializing [15415.658216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15415.681667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15420.042620] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15430.145911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15430.219668] Lustre: 491224:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15430.258410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15430.260714] Lustre: Skipped 1 previous similar message [15430.330309] Lustre: lustre-MDT0001: new disk, initializing [15430.435966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15430.483716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15430.494579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15434.363378] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15440.992899] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15445.334844] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15453.364967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15453.655413] Lustre: lustre-OST0000: new disk, initializing [15453.664538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15453.670745] Lustre: 492966:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15453.777239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15460.965994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15460.979313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15461.088218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15461.344883] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15468.270503] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15473.366338] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15473.553704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15477.732209] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15477.971344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15481.831893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15481.850741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15481.863888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15481.866041] LustreError: Skipped 1 previous similar message [15481.884066] Lustre: Skipped 1 previous similar message [15486.958810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15492.073852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15492.089407] Lustre: Skipped 2 previous similar messages [15494.112867] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15494.291748] Lustre: server umount lustre-OST0000 complete [15502.306533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15502.307450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15502.310884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15502.325657] Lustre: Skipped 2 previous similar messages [15504.080053] Lustre: server umount lustre-MDT0000 complete [15508.341458] LustreError: 491215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241323 with bad export cookie 17506202151552052226 [15508.343835] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15508.351751] LustreError: 491215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15508.685110] Lustre: server umount lustre-MDT0001 complete [15533.631649] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [15542.678492] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [15570.138896] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15322 sec ========= 01:16:24 (1781241384) [15571.934383] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:16:25 (1781241385) === [15575.572515] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:16:29 (1781241389) === [15616.817985] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [15619.604822] Key type lgssc unregistered [15619.871872] LNet: 499413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15619.880566] LNetError: 499413:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15620.936158] LNet: Removed LNI 192.168.201.115@tcp [15622.180965] Key type .llcrypt unregistered [15622.183736] Key type ._llcrypt unregistered