[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476022662 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2512MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003349] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013063] *** VALIDATE selinux *** [ 0.021734] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025766] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029128] *** VALIDATE tmpfs *** [ 0.030461] *** VALIDATE proc *** [ 0.031237] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033218] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035032] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.040964] debug: unmapping init [mem 0xffffffffa3259000-0xffffffffa3260fff] [ 0.042166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043681] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051327] rcu: Hierarchical SRCU implementation. [ 0.053397] smp: Bringing up secondary CPUs ... [ 0.054588] x86: Booting SMP configuration: [ 0.055019] .... node #0, CPUs: #1 #2 #3 [ 0.058295] smp: Brought up 1 node, 4 CPUs [ 0.060008] smpboot: Max logical packages: 1 [ 0.060886] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234140] node 0 deferred pages initialised in 171ms [ 0.238121] devtmpfs: initialized [ 0.239279] x86/mm: Memory block size: 128MB [ 0.243557] gcov: version magic: 0x41383552 [ 0.245264] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252299] pinctrl core: initialized pinctrl subsystem [ 0.255202] [ 0.255892] ************************************************************* [ 0.258014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261013] ** ** [ 0.263017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.265011] ** ** [ 0.267017] ** This means that this kernel is built to expose internal ** [ 0.269018] ** IOMMU data structures, which may compromise security on ** [ 0.271012] ** your system. ** [ 0.274014] ** ** [ 0.276027] ** If you see this message and you are not debugging the ** [ 0.278030] ** kernel, report this immediately to your vendor! ** [ 0.281013] ** ** [ 0.283015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285012] ************************************************************* [ 0.289502] NET: Registered protocol family 16 [ 0.291526] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.294078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.297088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.302023] cpuidle: using governor menu [ 0.303669] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.306720] PCI: Using configuration type 1 for base access [ 0.309162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.320165] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.323033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.327691] cryptd: max_cpu_qlen set to 1000 [ 0.330294] ACPI: Added _OSI(Module Device) [ 0.332021] ACPI: Added _OSI(Processor Device) [ 0.334020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.336021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.341211] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.349363] ACPI: Interpreter enabled [ 0.351084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.353020] ACPI: Using IOAPIC for interrupt routing [ 0.355147] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.360532] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.372174] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.374064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.378027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.382117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.388620] acpiphp: Slot [2] registered [ 0.390244] acpiphp: Slot [5] registered [ 0.392208] acpiphp: Slot [6] registered [ 0.393260] acpiphp: Slot [7] registered [ 0.395227] acpiphp: Slot [8] registered [ 0.397180] acpiphp: Slot [9] registered [ 0.399272] acpiphp: Slot [10] registered [ 0.402229] acpiphp: Slot [3] registered [ 0.404121] acpiphp: Slot [4] registered [ 0.405193] acpiphp: Slot [11] registered [ 0.407214] acpiphp: Slot [12] registered [ 0.409096] acpiphp: Slot [13] registered [ 0.411137] acpiphp: Slot [14] registered [ 0.413163] acpiphp: Slot [15] registered [ 0.415125] acpiphp: Slot [16] registered [ 0.417122] acpiphp: Slot [17] registered [ 0.418113] acpiphp: Slot [18] registered [ 0.420127] acpiphp: Slot [19] registered [ 0.422163] acpiphp: Slot [20] registered [ 0.424120] acpiphp: Slot [21] registered [ 0.426108] acpiphp: Slot [22] registered [ 0.427137] acpiphp: Slot [23] registered [ 0.429146] acpiphp: Slot [24] registered [ 0.431185] acpiphp: Slot [25] registered [ 0.433159] acpiphp: Slot [26] registered [ 0.434112] acpiphp: Slot [27] registered [ 0.436121] acpiphp: Slot [28] registered [ 0.438098] acpiphp: Slot [29] registered [ 0.439090] acpiphp: Slot [30] registered [ 0.441112] acpiphp: Slot [31] registered [ 0.443088] PCI host bridge to bus 0000:00 [ 0.444020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.448034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.451041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.453034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.456041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.459038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.461293] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.465257] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.468341] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.480014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.485132] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.488020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.490028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.493017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.496783] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.499908] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.503057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.506689] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.511016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.525021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.531020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.537452] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.545017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.552018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.569015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.579305] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.589021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.598020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.625031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.637536] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.654015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.667016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.696020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.705907] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.712022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.718031] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.738025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.753399] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.762016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.772017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.795020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.807797] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.817020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.827022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.845022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.859412] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.861364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.863335] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.866356] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.868197] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.872275] iommu: Default domain type: Passthrough [ 0.875399] SCSI subsystem initialized [ 0.876146] ACPI: bus type USB registered [ 0.878084] usbcore: registered new interface driver usbfs [ 0.880080] usbcore: registered new interface driver hub [ 0.881085] usbcore: registered new device driver usb [ 0.883142] pps_core: LinuxPPS API ver. 1 registered [ 0.884009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.886055] PTP clock support registered [ 0.888184] EDAC MC: Ver: 3.0.0 [ 0.890068] PCI: Using ACPI for IRQ routing [ 0.891514] NetLabel: Initializing [ 0.893014] NetLabel: domain hash size = 128 [ 0.895012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.898079] NetLabel: unlabeled traffic allowed by default [ 0.901045] vgaarb: loaded [ 0.903288] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.904013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.910861] clocksource: Switched to clocksource kvm-clock [ 1.021682] VFS: Disk quotas dquot_6.6.0 [ 1.022927] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.025130] *** VALIDATE ramfs *** [ 1.026165] *** VALIDATE hugetlbfs *** [ 1.027504] pnp: PnP ACPI init [ 1.029790] pnp: PnP ACPI: found 6 devices [ 1.048528] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.051309] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.053535] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.055660] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.058252] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.060870] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.063857] NET: Registered protocol family 2 [ 1.066188] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.070963] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.074815] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.079377] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.082271] TCP: Hash tables configured (established 65536 bind 65536) [ 1.084815] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.087445] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.089814] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.092292] NET: Registered protocol family 1 [ 1.094425] RPC: Registered named UNIX socket transport module. [ 1.100128] RPC: Registered udp transport module. [ 1.102206] RPC: Registered tcp transport module. [ 1.104727] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.107084] NET: Registered protocol family 44 [ 1.109083] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.111862] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.115448] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.118138] PCI: CLS 0 bytes, default 64 [ 1.120109] Unpacking initramfs... [ 2.556878] debug: unmapping init [mem 0xffffa0c3bcc54000-0xffffa0c3bffbffff] [ 2.560784] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.563153] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.566269] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.058639] Initialise system trusted keyrings [ 3.060781] Key type blacklist registered [ 3.062792] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.073329] zbud: loaded [ 3.077296] *** VALIDATE nfs *** [ 3.079365] *** VALIDATE nfs4 *** [ 3.081720] pstore: using deflate compression [ 3.086322] Platform Keyring initialized [ 3.181902] NET: Registered protocol family 38 [ 3.183322] Key type asymmetric registered [ 3.184401] Asymmetric key parser 'x509' registered [ 3.186431] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.190050] io scheduler mq-deadline registered [ 3.192149] io scheduler kyber registered [ 3.194209] io scheduler bfq registered [ 3.197108] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.200430] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.203551] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.206791] ACPI: Power Button [PWRF] [ 3.212387] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.219475] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.232311] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.238497] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.253857] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.283246] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.312169] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.317122] Non-volatile memory driver v1.3 [ 3.318736] Linux agpgart interface v0.103 [ 3.346180] virtio_blk virtio1: [vda] 147088 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.349310] vda: detected capacity change from 0 to 75309056 [ 3.364253] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.367255] vdb: detected capacity change from 0 to 1073741824 [ 3.385865] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.389177] vdc: detected capacity change from 0 to 2621440000 [ 3.404074] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.407234] vdd: detected capacity change from 0 to 2621440000 [ 3.421297] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.424190] vde: detected capacity change from 0 to 4294967296 [ 3.438160] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.441204] vdf: detected capacity change from 0 to 4294967296 [ 3.449268] libphy: Fixed MDIO Bus: probed [ 3.454461] usbcore: registered new interface driver usbserial_generic [ 3.456603] usbserial: USB Serial support registered for generic [ 3.458469] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.462573] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.464788] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.467569] mousedev: PS/2 mouse device common for all mice [ 3.471499] rtc_cmos 00:05: RTC can wake from S4 [ 3.474078] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.476878] rtc_cmos 00:05: registered as rtc0 [ 3.480853] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.483566] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.484559] intel_pstate: CPU model not supported [ 3.489994] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.492631] hid: raw HID events driver (C) Jiri Kosina [ 3.497329] usbcore: registered new interface driver usbhid [ 3.499995] usbhid: USB HID core driver [ 3.501727] drop_monitor: Initializing network drop monitor service [ 3.504219] Initializing XFRM netlink socket [ 3.506486] NET: Registered protocol family 10 [ 3.509314] Segment Routing with IPv6 [ 3.510607] NET: Registered protocol family 17 [ 3.512623] mpls_gso: MPLS GSO support [ 3.517259] RAS: Correctable Errors collector initialized. [ 3.519477] AVX version of gcm_enc/dec engaged. [ 3.521139] AES CTR mode by8 optimization enabled [ 3.589167] sched_clock: Marking stable (3589145955, 0)->(4525475715, -936329760) [ 3.593378] registered taskstats version 1 [ 3.595389] Loading compiled-in X.509 certificates [ 3.597607] zswap: loaded using pool lzo/zbud [ 3.617625] Key type big_key registered [ 3.629179] Key type encrypted registered [ 3.631319] ima: No TPM chip found, activating TPM-bypass! [ 3.633556] ima: Allocated hash algorithm: sha1 [ 3.635501] ima: No architecture policies found [ 3.637682] evm: Initialising EVM extended attributes: [ 3.639941] evm: security.selinux [ 3.641396] evm: security.ima [ 3.642619] evm: security.capability [ 3.643979] evm: HMAC attrs: 0x1 [ 3.646471] rtc_cmos 00:05: setting system clock to 2026-08-27 19:59:55 UTC (1787860795) [ 3.652210] debug: unmapping init [mem 0xffffffffa4203000-0xffffffffa43fffff] [ 3.655318] debug: unmapping init [mem 0xffffffffa2f82000-0xffffffffa3258fff] [ 3.664062] Write protecting the kernel read-only data: 28672k [ 3.667663] debug: unmapping init [mem 0xffffffffa1603000-0xffffffffa17fffff] [ 3.670664] debug: unmapping init [mem 0xffffffffa1f14000-0xffffffffa1ffffff] [ 3.705610] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.716105] systemd[1]: Detected virtualization kvm. [ 3.718091] systemd[1]: Detected architecture x86-64. [ 3.720141] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.745236] systemd[1]: No hostname configured. [ 3.747049] systemd[1]: Set hostname to . [ 3.749209] random: systemd: uninitialized urandom read (16 bytes read) [ 3.751875] systemd[1]: Initializing machine ID from random generator. [ 3.788236] random: ln: uninitialized urandom read (6 bytes read) [ 3.867445] random: systemd: uninitialized urandom read (16 bytes read) [ 3.871538] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.879176] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.883349] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.455574] device-mapper: uevent: version 1.0.3 [ 4.457392] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.067718] virtio_net virtio0 ens2: renamed from eth0 [ 5.098014] random: fast init done [ 5.123588] scsi host0: ata_piix [ 5.137122] scsi host1: ata_piix [ 5.138449] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.140446] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.342488] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.148773] random: crng init done [ 10.150394] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.380217] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.443624] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.676404] SELinux: Disabled at runtime. [ 11.740060] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.750379] systemd[1]: Detected virtualization kvm. [ 11.752456] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.216755] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.220027] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.225044] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.229398] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.234143] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.242841] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.250610] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 12.290935] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.676179] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.890459] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.931314] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.044278] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.057420] EDAC sbridge: Ver: 1.1.2 [ 14.682954] Key type dns_resolver registered [ 14.966156] NFS: Registering the id_resolver key type [ 14.968502] Key type id_resolver registered [ 14.969754] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-server login: [ 53.379196] libcfs: loading out-of-tree module taints kernel. [ 53.441878] Key type ._llcrypt registered [ 53.444805] Key type .llcrypt registered [ 53.538348] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 75.226110] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 76.881970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.897626] alg: No test for adler32 (adler32-zlib) [ 78.291858] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 79.297913] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 81.031159] Key type lgssc registered [ 82.459269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.806781] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 103.856051] hrtimer: interrupt took 4042896 ns [ 141.185562] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 153.684779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 153.728289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.193387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.259677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.352589] Lustre: lustre-MDT0000: new disk, initializing [ 155.419037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.446598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.533515] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.234752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.397944] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.466654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.474429] Lustre: Skipped 1 previous similar message [ 177.611145] Lustre: lustre-MDT0001: new disk, initializing [ 177.688990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.731763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.759874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.877094] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.018739] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.694293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.097355] Lustre: lustre-OST0000: new disk, initializing [ 199.099959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.121673] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.235470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.295203] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.444188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.470165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.643190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.692096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.879789] Lustre: lustre-OST0001: new disk, initializing [ 220.882975] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.886457] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.935903] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.364919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 227.383563] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 227.457195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.293147] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.480388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.575690] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.160742] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.850627] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing yml_node [ 270.844959] Lustre: DEBUG MARKER: Client: 2.17.57.120 [ 273.994978] Lustre: DEBUG MARKER: MDS: 2.17.57.120 [ 276.558244] Lustre: DEBUG MARKER: OSS: 2.17.57.120 [ 278.346730] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 16:04:28 EDT 2026 [ 294.739994] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 296.507266] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 314.337550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.339483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.361377] Lustre: Skipped 2 previous similar messages [ 314.366594] Lustre: Skipped 3 previous similar messages [ 318.913387] Lustre: server umount lustre-MDT0000 complete [ 322.753597] LustreError: 10345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861114 with bad export cookie 17886339658871127554 [ 322.763609] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.767744] LustreError: 10345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 324.584050] LustreError: 6528:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.584888] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.606318] LustreError: 6528:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 324.634085] Lustre: Skipped 1 previous similar message [ 324.646066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.650990] Lustre: Skipped 1 previous similar message [ 327.591073] LustreError: 6529:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.592907] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 327.628646] Lustre: Skipped 1 previous similar message [ 332.704670] LustreError: 6527:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.706655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.718331] LustreError: 6527:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 332.737156] Lustre: Skipped 1 previous similar message [ 337.375349] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 337.692838] Lustre: server umount lustre-MDT0001 complete [ 341.553438] Lustre: server umount lustre-OST0000 complete [ 345.861634] Lustre: server umount lustre-OST0001 complete [ 353.829764] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 364.115969] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 400.353558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.634735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 400.694100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 400.787847] Lustre: lustre-MDT0000: new disk, initializing [ 400.844365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.863687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 405.708210] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 417.322276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.491362] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.546787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.549311] Lustre: Skipped 1 previous similar message [ 417.617731] Lustre: lustre-MDT0001: new disk, initializing [ 417.757291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.816988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 417.832236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 422.171713] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 432.156409] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 437.944606] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 444.943321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 445.233871] Lustre: lustre-OST0000: new disk, initializing [ 445.238723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 445.251856] Lustre: 17053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 445.348420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.619070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 446.631559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 446.737952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 451.389813] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 460.446199] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 464.781730] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.955431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 469.319990] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.526099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 472.544794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 472.546376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.554453] LustreError: Skipped 1 previous similar message [ 472.566070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.576330] Lustre: Skipped 2 previous similar messages [ 472.596415] Lustre: Skipped 1 previous similar message [ 477.571172] Lustre: server umount lustre-OST0000 complete [ 482.787497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.796288] Lustre: Skipped 1 previous similar message [ 482.797413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 482.802596] Lustre: Skipped 1 previous similar message [ 495.071325] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 495.323293] Lustre: server umount lustre-MDT0000 complete [ 498.147912] LustreError: 15097:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.172830] LustreError: 15097:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 498.878168] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861290 with bad export cookie 17886339658871129087 [ 498.884027] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 498.889621] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 499.466423] Lustre: server umount lustre-MDT0001 complete [ 503.460051] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:08:13 (1787861293) [ 505.250864] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 506.644976] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:08:16 (1787861296) [ 512.586432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.091553] LustreError: 19463:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.178178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.511524] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 522.729242] LustreError: 19464:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.753379] LustreError: 19464:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 525.418095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.317485] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 536.442270] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 541.485509] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 547.960988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 554.880365] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 562.684980] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 570.848559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 570.855062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.888762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 570.894794] Lustre: Skipped 4 previous similar messages [ 572.902616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 576.595354] Lustre: server umount lustre-OST0000 complete [ 582.121956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 582.133687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.144420] Lustre: Skipped 1 previous similar message [ 594.399639] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 594.559538] Lustre: server umount lustre-MDT0000 complete [ 597.474950] LustreError: 19459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.491350] LustreError: 19459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 597.974978] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861389 with bad export cookie 17886339658871129969 [ 597.980842] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.985264] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 598.267781] Lustre: server umount lustre-MDT0001 complete [ 607.489807] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 610.001648] Key type lgssc unregistered [ 610.245685] LNet: 22998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.258353] LNetError: 22998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.289176] LNet: Removed LNI 192.168.203.116@tcp [ 611.188141] Key type .llcrypt unregistered [ 611.191457] Key type ._llcrypt unregistered [ 633.517991] Key type ._llcrypt registered [ 633.521648] Key type .llcrypt registered [ 633.613482] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 634.845277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 634.866335] alg: No test for adler32 (adler32-zlib) [ 635.894346] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 636.222938] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 637.927672] Key type lgssc registered [ 639.267723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 647.972796] loop: module loaded [ 648.065083] loop0: detected capacity change from 0 to 204800000 [ 648.368817] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.384920] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.689402] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.648268] loop0: detected capacity change from 0 to 204800000 [ 654.833789] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.843284] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.874684] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.285394] loop0: detected capacity change from 0 to 204800000 [ 665.884850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 665.917927] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.240676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 667.302937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 667.401097] Lustre: lustre-MDT0000: new disk, initializing [ 667.515744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.534151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 672.567182] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 681.870732] loop1: detected capacity change from 0 to 204800000 [ 687.451982] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 687.778396] Lustre: lustre-OST07c6: new disk, initializing [ 687.783255] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 687.788941] Lustre: Skipped 1 previous similar message [ 687.792645] Lustre: 26188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 687.868666] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 694.234170] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 694.862770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 694.875933] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 694.895919] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 698.939728] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 698.954878] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 698.964946] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 698.979596] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 698.993444] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 698.999811] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 718.842787] Lustre: 25175:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 718.849292] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 718.853603] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.857381] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 718.863669] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 718.868202] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 723.691991] Lustre: 25176:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 723.702339] Lustre: 25176:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 723.708219] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 723.717827] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.724908] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 723.736693] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.747639] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 723.758932] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.765751] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 723.776935] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.787044] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 723.796255] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.021454] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 727.029780] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 727.034499] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 727.043846] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.054568] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.065087] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.068877] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 727.077624] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.087627] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 727.092950] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.097853] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.103274] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.877193] Lustre: 25175:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 731.890300] Lustre: 25175:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 731.899168] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 731.913509] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.921859] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 731.933333] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.939856] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 731.952555] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.963226] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 731.971292] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.981469] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 731.991263] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.518054] Lustre: 25176:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 743.535954] Lustre: 25176:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages [ 743.547249] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 743.557684] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 743.567370] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 743.576924] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 743.583528] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 743.600033] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 743.612096] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 743.617511] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 743.628028] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 743.638021] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.086769] Lustre: server umount lustre-OST07c6 complete [ 815.288742] Lustre: server umount lustre-MDT0000 complete [ 834.512457] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 837.424795] Key type lgssc unregistered [ 837.782056] LNet: 28752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.791610] LNetError: 28752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.834348] LNet: Removed LNI 192.168.203.116@tcp [ 838.774600] Key type .llcrypt unregistered [ 838.777175] Key type ._llcrypt unregistered [ 858.798554] Key type ._llcrypt registered [ 858.800996] Key type .llcrypt registered [ 858.957028] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 859.853946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.868039] alg: No test for adler32 (adler32-zlib) [ 860.924206] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 861.149106] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 862.808966] Key type lgssc registered [ 863.925354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 871.886562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 871.920628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.493576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.171254] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 883.680525] LustreError: 30031:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.717130] LustreError: 30031:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 888.057406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.380804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.397383] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 902.628653] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.704172] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 917.710205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 918.074044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.086883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 924.350404] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 932.480656] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 949.753740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 949.766211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.780745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.488671] Lustre: server umount lustre-OST0000 complete [ 958.945070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.946646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 958.957575] Lustre: Skipped 1 previous similar message [ 958.981049] Lustre: Skipped 1 previous similar message [ 961.956996] Lustre: server umount lustre-MDT0000 complete [ 964.070031] LustreError: 30031:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.085496] LustreError: 30031:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 965.787068] LustreError: 32728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861757 with bad export cookie 7888780272135366810 [ 965.796970] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 965.806208] LustreError: 32728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 975.504923] Lustre: server umount lustre-MDT0001 complete [ 988.389220] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 991.444965] Key type lgssc unregistered [ 991.840544] LNet: 33773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.863523] LNetError: 33773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.886934] LNet: Removed LNI 192.168.203.116@tcp [ 992.821543] Key type .llcrypt unregistered [ 992.825059] Key type ._llcrypt unregistered [ 1011.419933] Key type ._llcrypt registered [ 1011.422669] Key type .llcrypt registered [ 1011.505371] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1012.294901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1012.320254] alg: No test for adler32 (adler32-zlib) [ 1013.496404] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1013.795709] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1015.495222] Key type lgssc registered [ 1016.580269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1024.379624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1024.399185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.924532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.519654] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1035.247563] LustreError: 35044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.279095] LustreError: 35044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1040.354693] LustreError: 35045:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.620411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.961993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1046.293274] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1053.207396] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1058.179525] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1064.125854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1064.463322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.344764] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1070.644835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1077.697102] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1091.040315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1091.051628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.061092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.067267] Lustre: Skipped 1 previous similar message [ 1091.093190] Lustre: Skipped 1 previous similar message [ 1094.857705] Lustre: server umount lustre-OST0000 complete [ 1101.280464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.281271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1101.290052] Lustre: Skipped 1 previous similar message [ 1106.305100] Lustre: server umount lustre-MDT0000 complete [ 1106.412527] LustreError: 35045:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.433958] LustreError: 35045:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1109.884522] LustreError: 35024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861901 with bad export cookie 2417663194057251495 [ 1109.906351] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1110.244914] Lustre: server umount lustre-MDT0001 complete [ 1122.672158] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1125.571433] Key type lgssc unregistered [ 1125.854282] LNet: 38580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.863233] LNetError: 38580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.882980] LNet: Removed LNI 192.168.203.116@tcp [ 1126.794477] Key type .llcrypt unregistered [ 1126.796236] Key type ._llcrypt unregistered [ 1150.005169] Key type ._llcrypt registered [ 1150.008806] Key type .llcrypt registered [ 1150.142349] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1150.993511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.045645] alg: No test for adler32 (adler32-zlib) [ 1152.037996] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1152.278236] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1153.951909] Key type lgssc registered [ 1154.832763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1163.095393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1163.114534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.851775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1169.787418] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1175.019051] LustreError: 39996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.064200] LustreError: 39996:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1180.056951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.129445] LustreError: 39997:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.531153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1185.927414] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1194.632476] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1201.375514] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1208.643586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1208.977174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.033736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1216.001859] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1223.745398] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1230.821147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.825172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.852142] Lustre: Skipped 1 previous similar message [ 1230.870464] Lustre: Skipped 2 previous similar messages [ 1231.998186] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1235.943335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.946557] Lustre: Skipped 2 previous similar messages [ 1236.872592] Lustre: server umount lustre-MDT0000 complete [ 1237.108182] LustreError: 40971:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.069504] LustreError: 39991:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.086600] LustreError: 39991:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1241.252600] LustreError: 39976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862033 with bad export cookie 10438890411876503163 [ 1241.275468] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.251297] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 1246.181074] LustreError: 39991:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.182604] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.205040] LustreError: 39991:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1246.237767] Lustre: Skipped 1 previous similar message [ 1247.343576] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 1247.358781] Lustre: Skipped 1 previous similar message [ 1247.667548] Lustre: server umount lustre-MDT0001 complete [ 1269.157461] Lustre: server umount lustre-OST0000 complete [ 1284.005437] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1286.982710] Key type lgssc unregistered [ 1287.338282] LNet: 43861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.347856] LNetError: 43861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.384443] LNet: Removed LNI 192.168.203.116@tcp [ 1288.313447] Key type .llcrypt unregistered [ 1288.321100] Key type ._llcrypt unregistered [ 1312.733831] Key type ._llcrypt registered [ 1312.737831] Key type .llcrypt registered [ 1312.833681] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1313.828743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1313.898384] alg: No test for adler32 (adler32-zlib) [ 1314.947725] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1315.185896] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1316.903180] Key type lgssc registered [ 1318.160949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1326.633541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1326.653191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.231615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.523401] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1338.876673] LustreError: 45277:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.896739] LustreError: 45277:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1341.543078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.797164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.615291] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1353.204278] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1359.794065] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1367.130370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1367.479879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.518725] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1373.705796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1381.039033] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1388.001834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1388.020829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.044215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1389.032860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1389.035176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1389.040767] Lustre: Skipped 1 previous similar message [ 1389.055399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.265034] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1393.278150] Lustre: Skipped 1 previous similar message [ 1398.386856] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1398.392935] Lustre: Skipped 2 previous similar messages [ 1401.311194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1401.462574] Lustre: server umount lustre-OST0000 complete [ 1418.719930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1418.724317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.735227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.743580] Lustre: Skipped 2 previous similar messages [ 1424.996753] Lustre: server umount lustre-MDT0000 complete [ 1428.851831] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862220 with bad export cookie 461806864309260579 [ 1428.861723] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1428.865143] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1443.295173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1443.523983] Lustre: server umount lustre-MDT0001 complete [ 1453.077415] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1455.702762] Key type lgssc unregistered [ 1456.085689] LNet: 48993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.095671] LNetError: 48993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.119292] LNet: Removed LNI 192.168.203.116@tcp [ 1457.101598] Key type .llcrypt unregistered [ 1457.103936] Key type ._llcrypt unregistered [ 1483.125835] Key type ._llcrypt registered [ 1483.128361] Key type .llcrypt registered [ 1483.222365] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1484.240756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1484.389254] alg: No test for adler32 (adler32-zlib) [ 1485.379851] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1485.652838] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1487.295196] Key type lgssc registered [ 1488.310861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1496.722319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1496.744608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.133783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.931568] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1507.301843] LustreError: 50409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.332393] LustreError: 50409:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1512.423894] LustreError: 50410:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.383706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.923801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1523.164873] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1532.228617] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1539.430239] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1546.006828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1546.476616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.186324] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1562.255187] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1567.997464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1572.308120] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1576.754668] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1590.639788] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2716h26m18s [ 1592.086388] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1597.928101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1597.943058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1597.960183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.074374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.088806] Lustre: Skipped 4 previous similar messages [ 1600.172784] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1600.190799] Lustre: Skipped 1 previous similar message [ 1601.396850] Lustre: server umount lustre-MDT0000 complete [ 1604.781383] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1605.611912] LustreError: 51138:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.635836] LustreError: 51138:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1611.146304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.273588] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1611.464930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.549654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1612.472816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1616.340457] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1620.045589] Lustre: DEBUG MARKER: Wait for df (36736) ... [ 1621.843399] Lustre: DEBUG MARKER: done [ 1627.108980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1627.120775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.146370] Lustre: Skipped 2 previous similar messages [ 1627.167357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.622622] Lustre: server umount lustre-OST0000 complete [ 1638.368316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.374624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1638.382720] Lustre: Skipped 1 previous similar message [ 1638.383929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.393988] LustreError: Skipped 1 previous similar message [ 1638.404619] Lustre: Skipped 4 previous similar messages [ 1642.827757] Lustre: server umount lustre-MDT0000 complete [ 1645.539979] LustreError: 50410:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.550601] LustreError: 50410:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1646.393401] LustreError: 50391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862438 with bad export cookie 11011041112274178481 [ 1646.400150] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1646.411498] LustreError: 50391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1646.848212] Lustre: server umount lustre-MDT0001 complete [ 1657.488270] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1660.157233] Key type lgssc unregistered [ 1660.469161] LNet: 55344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.475292] LNetError: 55344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.548520] LNet: Removed LNI 192.168.203.116@tcp [ 1662.452332] Key type .llcrypt unregistered [ 1662.453868] Key type ._llcrypt unregistered [ 1704.027421] Key type ._llcrypt registered [ 1704.029279] Key type .llcrypt registered [ 1704.131031] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1704.937529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1705.022093] alg: No test for adler32 (adler32-zlib) [ 1706.068728] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1706.328238] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1708.015163] Key type lgssc registered [ 1709.066114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1715.479644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1715.499649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.709653] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1716.739382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1716.906134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1720.926756] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1729.250287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.431283] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1729.482907] Lustre: 57554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1729.785862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1735.303578] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1743.783248] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1750.626620] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1756.849133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1757.029669] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1757.304707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.255144] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1767.089137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1770.871516] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1777.954729] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2716h29m27s [ 1779.737598] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1785.966231] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1788.626945] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1788.693473] LustreError: 57561:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1809.087373] Lustre: lustre-MDT0000: Client b8eadb22-a62e-4eb6-b1c9-70cf3c15e4e2 (at 192.168.203.16@tcp) reconnecting [ 1828.791205] LustreError: 57561:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1831.062826] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1833.407128] Lustre: DEBUG MARKER: done [ 1843.695204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1843.707099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.725184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1845.877265] Lustre: server umount lustre-OST0000 complete [ 1852.900058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1852.914560] LustreError: Skipped 1 previous similar message [ 1852.922189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.935137] Lustre: Skipped 1 previous similar message [ 1852.944484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1852.948788] Lustre: Skipped 2 previous similar messages [ 1853.991701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1854.005735] Lustre: Skipped 2 previous similar messages [ 1856.032795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.045539] Lustre: Skipped 1 previous similar message [ 1856.251264] Lustre: server umount lustre-MDT0000 complete [ 1859.465057] LustreError: 57544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862651 with bad export cookie 4439601669550012996 [ 1859.467696] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1859.485203] LustreError: 57544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1859.846859] Lustre: server umount lustre-MDT0001 complete [ 1869.725196] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1872.474522] Key type lgssc unregistered [ 1872.764268] LNet: 61886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.776428] LNetError: 61886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.811478] LNet: Removed LNI 192.168.203.116@tcp [ 1873.630811] Key type .llcrypt unregistered [ 1873.633708] Key type ._llcrypt unregistered [ 1916.863432] Key type ._llcrypt registered [ 1916.868224] Key type .llcrypt registered [ 1916.991205] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1918.267112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1918.297510] alg: No test for adler32 (adler32-zlib) [ 1919.355343] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 1919.623774] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1921.319580] Key type lgssc registered [ 1922.638234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1925.916575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1925.934913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.151782] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1927.176940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1927.320193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.126187] Lustre: Failing over lustre-MDT0000 [ 1928.334144] Lustre: server umount lustre-MDT0000 complete [ 1934.094783] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:32:04 (1787862724) [ 1940.309339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.702804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.738413] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1952.602202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.686429] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1952.707985] Lustre: 64689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1952.946202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1957.329218] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1965.307764] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1971.517244] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1977.662603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1977.861345] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1978.088110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.199791] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1986.249558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1991.561901] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1997.766769] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2003.380335] Lustre: server umount lustre-MDT0000 complete [ 2004.449977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2004.456241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.914224] LustreError: 65426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862798 with bad export cookie 11292341668326404956 [ 2006.918743] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.010402] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.028043] Lustre: Skipped 1 previous similar message [ 2007.040637] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2009.400912] Lustre: server umount lustre-MDT0001 complete [ 2013.755616] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2019.676676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2028.414258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.520994] LustreError: 63610:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000414769ba x1874709930088448/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2031.807992] LustreError: 66539:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.818525] LustreError: 66539:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2031.867709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.949551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2036.074330] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2037.217697] LustreError: 66540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.350300] LustreError: 66539:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.181830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.593869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2045.620052] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2049.069128] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2055.725549] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2061.262831] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2069.984474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2070.000370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.024557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.524196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.536451] Lustre: Skipped 2 previous similar messages [ 2076.643667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.650106] Lustre: Skipped 2 previous similar messages [ 2081.772123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.780728] Lustre: Skipped 2 previous similar messages [ 2082.272075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2082.496338] Lustre: server umount lustre-MDT0000 complete [ 2085.917868] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862877 with bad export cookie 11292341668326407231 [ 2085.919096] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.932449] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2086.314919] Lustre: server umount lustre-MDT0001 complete [ 2092.474663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2102.315242] Lustre: 63613:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862878/real 1787862878] req@0000000016c65b10 x1874709930115584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787862894 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2102.344553] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.364197] Lustre: Skipped 2 previous similar messages [ 2102.718811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.463152] Lustre: 63613:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862882/real 1787862882] req@00000000a561910a x1874709930115712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787862898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2110.562120] LustreError: 63610:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000025dbfdf3 x1874709930116480/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2110.799289] LustreError: 66539:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.880778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.938438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2111.455071] Lustre: 63612:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862887/real 1787862887] req@000000007c0078cd x1874709930116096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787862903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2111.481913] Lustre: 63612:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2111.915074] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2111.918781] Lustre: Skipped 1 previous similar message [ 2114.659927] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2118.048819] Lustre: 63614:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862893/real 1787862893] req@00000000c84cc144 x1874709930116224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787862909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2118.116647] LustreError: 71142:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.138064] LustreError: 71142:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2122.429342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.409104] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2128.373818] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2135.331739] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2141.401092] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2147.845448] Lustre: server umount lustre-MDT0000 complete [ 2148.320929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2148.326728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.339453] LustreError: 66540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.911346] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862942 with bad export cookie 11292341668326408610 [ 2150.916106] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.923990] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2153.218814] Lustre: server umount lustre-MDT0001 complete [ 2159.002638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2161.903254] Lustre: DEBUG MARKER: files compared the same [ 2173.983316] Lustre: server umount lustre-OST0000 complete [ 2187.961142] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2190.414387] Key type lgssc unregistered [ 2190.691536] LNet: 74423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.697850] LNetError: 74423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.716748] LNet: Removed LNI 192.168.203.116@tcp [ 2191.543864] Key type .llcrypt unregistered [ 2191.545848] Key type ._llcrypt unregistered [ 2211.658272] Key type ._llcrypt registered [ 2211.659964] Key type .llcrypt registered [ 2211.801559] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2212.934404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.981559] alg: No test for adler32 (adler32-zlib) [ 2214.070425] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2214.308410] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2215.953796] Key type lgssc registered [ 2217.070825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2224.524866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2224.544906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.098274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.639085] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 2235.365597] LustreError: 75839:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.383442] LustreError: 75839:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2239.854239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.221202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2244.888542] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 2252.061191] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2257.582503] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2263.940961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2264.122395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.184507] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 2270.245857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2275.682493] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2285.535626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2285.535882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.544089] LustreError: Skipped 1 previous similar message [ 2285.545727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2285.559065] Lustre: Skipped 1 previous similar message [ 2285.571620] Lustre: Skipped 1 previous similar message [ 2289.622555] Lustre: server umount lustre-OST0000 complete [ 2295.785337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.792899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.816567] Lustre: Skipped 1 previous similar message [ 2295.832420] Lustre: Skipped 1 previous similar message [ 2299.870868] Lustre: server umount lustre-MDT0000 complete [ 2300.900698] LustreError: 75835:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.923360] LustreError: 75835:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2303.837638] LustreError: 75817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863095 with bad export cookie 14620311781374929063 [ 2303.847049] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.079032] Lustre: server umount lustre-MDT0001 complete [ 2315.697855] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2318.706535] Key type lgssc unregistered [ 2318.997519] LNet: 79373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.009897] LNetError: 79373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.030635] LNet: Removed LNI 192.168.203.116@tcp [ 2319.899514] Key type .llcrypt unregistered [ 2319.902751] Key type ._llcrypt unregistered [ 2347.574574] Key type ._llcrypt registered [ 2347.579421] Key type .llcrypt registered [ 2347.748787] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2349.149078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2349.220072] alg: No test for adler32 (adler32-zlib) [ 2350.329657] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2350.692832] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2352.384228] Key type lgssc registered [ 2354.197968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2362.048380] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2362.065703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2377.631426] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2382.752740] LustreError: 80765:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 2409.375201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.318843] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2423.967339] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2431.437903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.539551] Lustre: *** cfs_fail_loc=706, val=0*** [ 2431.882891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.950886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2436.625765] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2437.096474] LustreError: 80790:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.133812] LustreError: 80790:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2442.212809] LustreError: 80791:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.244016] LustreError: 80790:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.022946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.397747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.051358] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2457.340599] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2463.847249] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2467.809433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2467.828356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.845908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2470.883689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2470.884806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2470.896794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.916937] Lustre: Skipped 1 previous similar message [ 2472.599757] Lustre: server umount lustre-OST0000 complete [ 2481.121843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.125419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2481.138632] Lustre: Skipped 1 previous similar message [ 2481.168109] Lustre: Skipped 1 previous similar message [ 2482.772186] Lustre: server umount lustre-MDT0000 complete [ 2486.241525] LustreError: 81855:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.256825] LustreError: 81855:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2487.274304] LustreError: 80771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863279 with bad export cookie 13494013816429833501 [ 2487.283714] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.298479] LustreError: 80771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2487.671738] Lustre: server umount lustre-MDT0001 complete [ 2498.905548] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2501.973841] Key type lgssc unregistered [ 2502.179741] LNet: 84383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2502.185312] LNetError: 84383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.235085] LNet: Removed LNI 192.168.203.116@tcp [ 2503.200873] Key type .llcrypt unregistered [ 2503.203476] Key type ._llcrypt unregistered [ 2532.674816] Key type ._llcrypt registered [ 2532.678234] Key type .llcrypt registered [ 2532.814490] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2534.212225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2534.332641] alg: No test for adler32 (adler32-zlib) [ 2535.507214] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2535.694186] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2537.311114] Key type lgssc registered [ 2538.706494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2546.760735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2546.783408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.966556] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2561.004908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.481239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2565.246523] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2566.626619] LustreError: 86496:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.659680] LustreError: 86496:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2571.749403] LustreError: 86500:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.770419] LustreError: 86500:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2573.030331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2573.472966] LustreError: 86499:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.508560] LustreError: 86499:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2573.529967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.913742] LustreError: 86500:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.932903] LustreError: 86500:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2580.349109] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2584.046605] LustreError: 86499:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.072561] LustreError: 86499:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2589.734932] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.764346] Lustre: MGS: Client 8e78c670-5f3f-4894-8dd8-8ea624eb7fa2 (at 0@lo) reconnecting [ 2589.800180] LustreError: 86510:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2590.149864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.257964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2595.404681] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2610.662444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.685519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2610.690685] Lustre: Skipped 1 previous similar message [ 2610.692152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2610.739960] Lustre: Skipped 1 previous similar message [ 2614.578361] Lustre: server umount lustre-OST0000 complete [ 2620.899080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.902382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.914775] Lustre: Skipped 1 previous similar message [ 2625.489973] Lustre: server umount lustre-MDT0000 complete [ 2626.024913] LustreError: 88292:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.055672] LustreError: 88292:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2629.954438] Lustre: server umount lustre-MDT0001 complete [ 2633.787230] Lustre: server umount MGS complete [ 2646.239923] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:43:55 (1787863435) [ 2675.150511] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2691.865306] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2731.190834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.695752] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2744.932864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.185343] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2745.247850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2745.398997] Lustre: lustre-MDT0001: new disk, initializing [ 2745.515294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2745.547821] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2745.556165] Lustre: 93696:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2745.576235] LustreError: 93696:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2749.949295] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2761.448590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.778629] Lustre: lustre-OST0000: new disk, initializing [ 2761.785022] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2761.793657] Lustre: 94461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2761.863319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.900454] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2768.928774] Lustre: 94756:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000f0d98fc x1874710576013312/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787863576 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2779.929323] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2787.863602] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2787.905457] Lustre: MGS: Client 06145e05-16c4-4066-9389-7435bc2a3e19 (at 0@lo) reconnecting [ 2787.992819] Lustre: 92965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2788.021194] LustreError: 93682:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2788.043724] LustreError: 93682:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2788.092985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2788.222591] Lustre: lustre-MDT0000: new disk, initializing [ 2788.436104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2788.477499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2788.512353] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2789.547031] Lustre: 95530:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002fe49082 x1874710576020352/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787863597 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2794.390776] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2794.758092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2794.779445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2794.817622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2798.649092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2798.669720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2820.582794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.583938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2820.587871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2820.596100] Lustre: Skipped 1 previous similar message [ 2820.618152] Lustre: Skipped 2 previous similar messages [ 2825.714914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2825.736445] Lustre: Skipped 1 previous similar message [ 2826.543258] Lustre: server umount lustre-OST0000 complete [ 2835.940411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.940802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2835.950675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.962769] Lustre: Skipped 1 previous similar message [ 2838.597842] Lustre: server umount lustre-MDT0000 complete [ 2841.059150] LustreError: 93671:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.082708] LustreError: 93671:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2850.105455] Lustre: server umount lustre-MDT0001 complete [ 2854.352442] Lustre: server umount MGS complete [ 2868.491519] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:47:37 (1787863657) [ 2887.485753] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2890.513575] Key type lgssc unregistered [ 2890.929642] LNet: 98037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2890.936417] LNetError: 98037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2890.953065] LNet: Removed LNI 192.168.203.116@tcp [ 2891.893667] Key type .llcrypt unregistered [ 2891.897661] Key type ._llcrypt unregistered [ 2911.456660] Key type ._llcrypt registered [ 2911.458545] Key type .llcrypt registered [ 2911.556633] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2912.792228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2912.838090] alg: No test for adler32 (adler32-zlib) [ 2913.859389] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 2914.152527] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2915.896223] Key type lgssc registered [ 2917.124635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2926.098697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2926.119679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.601482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.606479] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2937.832044] LustreError: 99260:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.842347] LustreError: 99260:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2940.937637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.354884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.844525] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2954.221523] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2960.862664] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2967.692622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2968.143630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2972.282543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2974.704444] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2982.558641] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2997.221761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.223418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2997.245282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3001.673980] Lustre: server umount lustre-OST0000 complete [ 3007.457664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.459884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3007.469750] Lustre: Skipped 2 previous similar messages [ 3007.472065] Lustre: Skipped 1 previous similar message [ 3011.723869] Lustre: server umount lustre-MDT0000 complete [ 3012.578387] LustreError: 99260:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.602773] LustreError: 99260:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3015.642787] LustreError: 99242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863807 with bad export cookie 13985388861445814032 [ 3015.651738] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3016.060511] Lustre: server umount lustre-MDT0001 complete [ 3027.621116] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3030.270779] Key type lgssc unregistered [ 3030.606655] LNet: 102816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3030.611796] LNetError: 102816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3030.626163] LNet: Removed LNI 192.168.203.116@tcp [ 3031.493339] Key type .llcrypt unregistered [ 3031.495000] Key type ._llcrypt unregistered [ 3048.069845] Key type ._llcrypt registered [ 3048.072133] Key type .llcrypt registered [ 3048.181342] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3049.534518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3049.663458] alg: No test for adler32 (adler32-zlib) [ 3050.808681] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 3051.210476] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3052.871603] Key type lgssc registered [ 3054.073941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3059.876518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3059.886492] LustreError: 103901:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3059.891656] LustreError: 103902:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3059.898632] LustreError: 103901:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3059.906380] LustreError: 103901:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3059.914575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.915655] LustreError: 103901:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3061.367789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.476071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.920192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3075.251098] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3083.016133] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing lsmod [ 3086.049345] LustreError: 105101:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3086.132800] LustreError: 105108:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3086.158972] LustreError: 105101:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4912 [ 3086.176123] LustreError: 105101:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3086.187697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3086.189897] LustreError: 105101:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3086.760096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.401079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3090.410287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.425992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.949715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.969069] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.987407] Lustre: Skipped 1 previous similar message [ 3095.080023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3095.090123] Lustre: Skipped 4 previous similar messages [ 3096.520947] Lustre: server umount lustre-MDT0000 complete [ 3100.131663] LustreError: 103939:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.146744] LustreError: 103939:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3100.820653] LustreError: 103915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863892 with bad export cookie 4551729094377618648 [ 3100.835985] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3100.836323] LustreError: 103915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3101.239848] LustreError: 104370:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000207aecd8 x1874711117059328/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3101.369440] Lustre: server umount lustre-MDT0001 complete [ 3114.752316] Lustre: server umount lustre-OST0000 complete [ 3122.385424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.882491] LustreError: 106173:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3122.961700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.635865] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3128.293375] LustreError: 106174:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.389790] LustreError: 106173:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.896614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.390750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3142.213113] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3151.148701] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3158.764037] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3165.868484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3166.230838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.301107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3172.208490] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3179.843619] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3192.289554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3192.295731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.312883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.331079] Lustre: Skipped 1 previous similar message [ 3192.349661] Lustre: Skipped 1 previous similar message [ 3197.412571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3197.430447] Lustre: Skipped 1 previous similar message [ 3197.712961] Lustre: server umount lustre-OST0000 complete [ 3202.528644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3207.652599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3207.664680] Lustre: Skipped 3 previous similar messages [ 3208.301162] Lustre: server umount lustre-MDT0000 complete [ 3212.548863] LustreError: 106153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864004 with bad export cookie 4551729094377619348 [ 3212.558960] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3212.560476] LustreError: 106153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3212.998984] Lustre: server umount lustre-MDT0001 complete [ 3226.550686] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3229.677642] Key type lgssc unregistered [ 3230.125807] LNet: 109714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3230.137648] LNetError: 109714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3230.156561] LNet: Removed LNI 192.168.203.116@tcp [ 3231.192059] Key type .llcrypt unregistered [ 3231.196907] Key type ._llcrypt unregistered [ 3256.823574] Key type ._llcrypt registered [ 3256.827882] Key type .llcrypt registered [ 3257.018211] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3258.336132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3258.388810] alg: No test for adler32 (adler32-zlib) [ 3259.432147] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 3259.690487] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3261.328141] Key type lgssc registered [ 3262.427223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3270.290401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3270.315340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.822575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.781403] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3281.382715] LustreError: 111134:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.402582] LustreError: 111134:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3284.843174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.220558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3289.104047] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3296.626225] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3302.427906] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3309.234289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3309.659619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.430730] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3316.842080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3324.469574] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3329.362484] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3332.652606] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3336.676246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3336.681708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.690351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3337.700715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3337.702748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3337.716934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.841056] LustreError: 112956:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.850013] LustreError: 112956:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3342.916791] Lustre: server umount lustre-OST0000 complete [ 3346.914411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3346.945336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.960671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3346.964027] Lustre: Skipped 1 previous similar message [ 3352.948300] Lustre: server umount lustre-MDT0000 complete [ 3353.055704] LustreError: 111134:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.075890] LustreError: 111134:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3356.742661] LustreError: 111868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864148 with bad export cookie 18427645936054544212 [ 3356.744240] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3356.752176] LustreError: 111868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3357.043716] Lustre: server umount lustre-MDT0001 complete [ 3368.354318] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3371.318327] Key type lgssc unregistered [ 3371.658653] LNet: 114775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3371.667639] LNetError: 114775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3371.685461] LNet: Removed LNI 192.168.203.116@tcp [ 3372.514399] Key type .llcrypt unregistered [ 3372.518321] Key type ._llcrypt unregistered [ 3389.951707] Key type ._llcrypt registered [ 3389.955632] Key type .llcrypt registered [ 3390.056528] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3391.303294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3391.321979] alg: No test for adler32 (adler32-zlib) [ 3392.344261] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 3392.586629] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3394.264502] Key type lgssc registered [ 3395.401334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3402.845105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3402.867769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.240045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.965716] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3414.496474] LustreError: 115995:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.520269] LustreError: 115995:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3416.436991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.925480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3421.337959] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3429.026511] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3434.789790] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3441.874112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3442.272253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.451833] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3449.461137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3457.070848] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3470.304053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3470.316371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.332574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3472.358803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3472.360519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3472.363125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3472.392817] Lustre: Skipped 1 previous similar message [ 3476.532271] Lustre: server umount lustre-OST0000 complete [ 3482.596819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.600922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.614734] Lustre: Skipped 1 previous similar message [ 3487.713083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3487.720590] Lustre: Skipped 2 previous similar messages [ 3492.833387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3492.846818] Lustre: Skipped 1 previous similar message [ 3495.391600] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3495.542654] Lustre: server umount lustre-MDT0000 complete [ 3497.953806] LustreError: 115995:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.971977] LustreError: 115995:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3498.733412] LustreError: 115976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864290 with bad export cookie 9814341541098156204 [ 3498.738911] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3498.740460] LustreError: 115976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3499.062963] Lustre: server umount lustre-MDT0001 complete [ 3511.412711] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3515.307811] Key type lgssc unregistered [ 3515.690873] LNet: 119554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3515.703632] LNetError: 119554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3515.725042] LNet: Removed LNI 192.168.203.116@tcp [ 3516.750106] Key type .llcrypt unregistered [ 3516.752383] Key type ._llcrypt unregistered [ 3540.803621] Key type ._llcrypt registered [ 3540.806335] Key type .llcrypt registered [ 3540.934110] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3542.163400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3542.257577] alg: No test for adler32 (adler32-zlib) [ 3543.258258] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 3543.428437] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3545.071938] Key type lgssc registered [ 3546.269558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3553.564244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3553.580637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.282234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.012655] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3565.540380] LustreError: 120973:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.566754] LustreError: 120973:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3570.424934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.658317] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3570.807534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3575.819388] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3583.052272] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3588.696249] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3594.985982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3595.344674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.444537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3601.476386] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3608.223804] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3614.687943] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3618.187728] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3618.194944] Lustre: Skipped 2 previous similar messages [ 3621.560750] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3621.602675] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3622.749657] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3622.754607] Lustre: Skipped 2 previous similar messages [ 3627.110754] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3627.122364] Lustre: Skipped 1 previous similar message [ 3633.639546] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.16@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3633.650197] Lustre: Skipped 2 previous similar messages [ 3635.985324] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.16@tcp 192.168.0.[2,10]@tcp [ 3644.210932] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.16/24@tcp in log lustre-MDT0000 [ 3644.219213] Lustre: Skipped 2 previous similar messages [ 3645.052458] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.16/24@tcp [ 3645.055961] Lustre: Skipped 1 previous similar message [ 3654.119264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3654.119787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.125122] LustreError: Skipped 2 previous similar messages [ 3654.132936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3656.676453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3656.677341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.259285] Lustre: server umount lustre-OST0000 complete [ 3666.915887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.942440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3666.950983] Lustre: Skipped 2 previous similar messages [ 3669.667419] Lustre: server umount lustre-MDT0000 complete [ 3672.033683] LustreError: 120973:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3672.047397] LustreError: 120973:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3673.737881] LustreError: 123601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864465 with bad export cookie 13750648024447834724 [ 3673.746981] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3673.754062] LustreError: 123601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3674.075902] Lustre: server umount lustre-MDT0001 complete [ 3685.556146] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3688.664458] Key type lgssc unregistered [ 3688.869992] LNet: 125691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3688.874638] LNetError: 125691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3689.900912] LNet: Removed LNI 192.168.203.116@tcp [ 3690.734505] Key type .llcrypt unregistered [ 3690.736347] Key type ._llcrypt unregistered [ 3720.101868] Key type ._llcrypt registered [ 3720.105822] Key type .llcrypt registered [ 3720.192780] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3720.885434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.904324] alg: No test for adler32 (adler32-zlib) [ 3721.850805] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 3722.032512] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3723.647302] Key type lgssc registered [ 3724.399883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3730.893720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3730.911760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.252117] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3732.271980] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.16/24@tcp [ 3732.280953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.610959] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3742.695631] LustreError: 127255:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.705146] LustreError: 127255:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3745.137869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.599502] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3745.640960] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.16/24@tcp [ 3745.682859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3749.831436] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3757.068201] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3763.631925] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3770.599655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3770.872373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.509920] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3778.057294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3783.639198] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3798.501867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3798.517598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3798.549522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3800.552942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3800.554727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3800.555062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.572515] Lustre: Skipped 1 previous similar message [ 3804.428976] Lustre: server umount lustre-OST0000 complete [ 3810.791795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.806308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3810.808053] Lustre: Skipped 1 previous similar message [ 3810.820756] Lustre: Skipped 1 previous similar message [ 3814.925798] Lustre: server umount lustre-MDT0000 complete [ 3815.906413] LustreError: 127250:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.928056] LustreError: 127250:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3818.621268] LustreError: 127997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864610 with bad export cookie 14235585530967212231 [ 3818.624380] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3818.625670] LustreError: 127997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3818.955174] Lustre: server umount lustre-MDT0001 complete [ 3828.637690] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3831.160548] Key type lgssc unregistered [ 3831.386449] LNet: 130847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3831.393135] LNetError: 130847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3832.424242] LNet: Removed LNI 192.168.203.116@tcp [ 3833.286163] Key type .llcrypt unregistered [ 3833.295087] Key type ._llcrypt unregistered [ 3856.133189] Key type ._llcrypt registered [ 3856.136046] Key type .llcrypt registered [ 3856.222529] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3867.806976] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3868.754592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3868.771700] alg: No test for adler32 (adler32-zlib) [ 3869.797303] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 3869.965560] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3871.647170] Key type lgssc registered [ 3872.756139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3905.436279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3905.477311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.713459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3906.774983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3906.886309] Lustre: lustre-MDT0000: new disk, initializing [ 3906.944314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3906.963300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3911.273603] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3921.538352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.652590] Lustre: 134229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3921.707958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3921.714216] Lustre: Skipped 1 previous similar message [ 3921.812594] Lustre: lustre-MDT0001: new disk, initializing [ 3921.939515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3921.985641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3921.999326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3926.873776] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3936.034428] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3941.807145] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3949.133644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3949.394651] Lustre: lustre-OST0000: new disk, initializing [ 3949.398820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3949.407404] Lustre: 136195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3949.491803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3955.766671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3955.775765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3955.870943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3956.331972] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3965.460331] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3969.250722] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3969.428941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3973.538965] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3973.762603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3976.160456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3976.164155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.176282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.676539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.677172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3976.687121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3981.588678] Lustre: server umount lustre-OST0000 complete [ 3986.915372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.917772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.927309] Lustre: Skipped 1 previous similar message [ 3991.445061] Lustre: server umount lustre-MDT0000 complete [ 3994.996777] LustreError: 134220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864786 with bad export cookie 18076199393658371908 [ 3994.998094] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3995.017896] LustreError: 134220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3995.608082] Lustre: server umount lustre-MDT0001 complete [ 4003.990756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.535907] LustreError: 138413:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.555490] LustreError: 138413:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4004.615235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.954194] LustreError: 138414:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4010.093361] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4015.078881] LustreError: 138413:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.485053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4020.189359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4024.750767] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4033.723304] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4039.469494] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4046.707940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4047.047740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.104863] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4060.740493] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4065.561865] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4065.749316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4070.427913] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4070.653471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4079.801049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.049681] Lustre: lustre-OST0001: new disk, initializing [ 4080.053358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4080.059138] Lustre: 141783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4080.210468] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4086.976638] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4089.456899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4089.475083] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4089.576527] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4095.563568] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4095.843404] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4100.604412] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4100.851039] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4106.839640] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f8750b22000.ost_server_uuid 50 [ 4108.584541] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f8750b22000.ost_server_uuid in IDLE state after 0 sec [ 4120.544668] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4120.549317] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.561097] LustreError: Skipped 1 previous similar message [ 4120.561905] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4120.561908] Lustre: Skipped 1 previous similar message [ 4120.605604] Lustre: Skipped 1 previous similar message [ 4124.006769] Lustre: server umount lustre-OST0001 complete [ 4125.673991] LustreError: 140235:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.694731] LustreError: 140235:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4130.790197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.799459] LustreError: 140235:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4130.806748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4130.806757] Lustre: Skipped 1 previous similar message [ 4130.819233] Lustre: Skipped 1 previous similar message [ 4130.865340] LustreError: 140235:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4133.874205] Lustre: server umount lustre-OST0000 complete [ 4137.952696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4141.025167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.032243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4141.053689] Lustre: Skipped 1 previous similar message [ 4141.056608] Lustre: Skipped 2 previous similar messages [ 4144.110216] Lustre: server umount lustre-MDT0000 complete [ 4146.150749] LustreError: 138410:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.170150] LustreError: 138410:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4147.788714] LustreError: 138393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864939 with bad export cookie 18076199393658372797 [ 4147.805196] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4148.016480] Lustre: server umount lustre-MDT0001 complete [ 4167.479575] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4170.698950] Key type lgssc unregistered [ 4171.039559] LNet: 144788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4171.049368] LNetError: 144788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4171.063867] LNet: Removed LNI 192.168.203.116@tcp [ 4172.161576] Key type .llcrypt unregistered [ 4172.164228] Key type ._llcrypt unregistered [ 4219.565667] Key type ._llcrypt registered [ 4219.569263] Key type .llcrypt registered [ 4219.661642] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4234.572965] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4236.186733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4236.249553] alg: No test for adler32 (adler32-zlib) [ 4237.354421] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 4237.600815] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4239.311150] Key type lgssc registered [ 4240.808935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4279.308906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4279.342782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.651423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4280.736714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4280.830786] Lustre: lustre-MDT0000: new disk, initializing [ 4280.930234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.948231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4286.264461] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4299.750876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.877432] Lustre: 148946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4299.922438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4299.926115] Lustre: Skipped 1 previous similar message [ 4300.020649] Lustre: lustre-MDT0001: new disk, initializing [ 4300.160519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4300.193285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4300.206142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4305.442563] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4314.741507] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4320.824542] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4327.612793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.948304] Lustre: lustre-OST0000: new disk, initializing [ 4327.951375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4327.960053] Lustre: 150916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4328.060512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.448018] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4336.729720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4336.741225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4336.939224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4345.190370] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4350.608209] Lustre: Failing over lustre-OST0000 [ 4350.762045] Lustre: server umount lustre-OST0000 complete [ 4368.863191] Lustre: 147169:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787865144/real 1787865144] req@00000000851afca7 x1874712360682368/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787865160 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4368.864131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.881136] Lustre: 147169:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4370.374506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4370.876989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.905423] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4373.897099] Lustre: 147169:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787865149/real 1787865149] req@0000000053a3bbec x1874712360684288/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787865165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4373.932736] Lustre: 147169:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4378.291899] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4378.736697] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4378.975750] Lustre: 147170:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787865154/real 1787865154] req@00000000013ba91e x1874712360685440/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787865170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4379.034062] Lustre: 147170:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4379.042137] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4379.042326] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4382.706314] Lustre: Failing over lustre-MDT0000 [ 4383.188180] Lustre: server umount lustre-MDT0000 complete [ 4383.202281] Lustre: 147170:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787865159/real 1787865159] req@000000009502ba58 x1874712360686464/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787865175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4383.237231] Lustre: 147170:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4384.225157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.247432] Lustre: Skipped 3 previous similar messages [ 4388.976864] LustreError: 148955:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.997688] LustreError: 148955:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4394.092949] LustreError: 151543:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.117133] LustreError: 151543:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4399.212466] LustreError: 150171:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.250083] LustreError: 150171:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4400.607785] Lustre: 147169:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787865176/real 1787865176] req@000000005c73788d x1874712360696192/t0(0) o400->MGC192.168.203.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787865192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4400.663821] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4403.099801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.333460] LustreError: 148953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.355430] LustreError: 148953:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4410.485204] LustreError: 148954:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4410.521815] LustreError: 148954:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4410.849207] LustreError: 147168:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000069e98528 x1874712360701440/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4411.163174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.238703] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4415.608409] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4415.959874] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4416.502521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4416.518132] Lustre: Skipped 1 previous similar message [ 4416.538683] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4416.593792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4425.185038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4425.217992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.233161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4426.727779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4426.740648] Lustre: Skipped 1 previous similar message [ 4431.091323] Lustre: server umount lustre-OST0000 complete [ 4436.961062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4436.968653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4436.981746] Lustre: Skipped 2 previous similar messages [ 4436.982183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4437.015000] Lustre: Skipped 1 previous similar message [ 4441.597756] Lustre: server umount lustre-MDT0000 complete [ 4442.082317] LustreError: 151543:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4442.102563] LustreError: 151543:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4445.216207] LustreError: 148939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865237 with bad export cookie 16237627708644848021 [ 4445.222108] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4445.234142] LustreError: 148939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4445.655947] Lustre: server umount lustre-MDT0001 complete [ 4457.729337] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4460.433131] Key type lgssc unregistered [ 4460.756777] LNet: 154576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4460.767808] LNetError: 154576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4461.798619] LNet: Removed LNI 192.168.203.116@tcp [ 4462.669222] Key type .llcrypt unregistered [ 4462.677949] Key type ._llcrypt unregistered [ 4500.785467] Key type ._llcrypt registered [ 4500.788934] Key type .llcrypt registered [ 4500.904992] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4502.259326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4502.282807] alg: No test for adler32 (adler32-zlib) [ 4503.346430] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 4503.645897] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4505.319152] Key type lgssc registered [ 4506.614412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4515.376552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4515.406982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.038089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.399588] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4526.565986] LustreError: 156099:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4526.613285] LustreError: 156099:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4531.690613] LustreError: 156100:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.027056] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4536.803420] LustreError: 156099:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.191820] LustreError: 156128:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4537.208498] LustreError: 156128:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4537.224657] LustreError: 156128:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4537.903857] Lustre: server umount lustre-MDT0000 complete [ 4552.957979] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4562.916439] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4602.100698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4602.398603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4602.475254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4602.594635] Lustre: lustre-MDT0000: new disk, initializing [ 4602.680956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.695491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4607.190497] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4617.787126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.918741] Lustre: 159433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4617.962983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4617.966530] Lustre: Skipped 1 previous similar message [ 4618.091991] Lustre: lustre-MDT0001: new disk, initializing [ 4618.194333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4618.239880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4618.257910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4622.515858] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4632.697894] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4640.884165] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4649.179211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4649.539432] Lustre: lustre-OST0000: new disk, initializing [ 4649.542856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4649.552449] Lustre: 161402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4649.721812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.962850] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4659.778953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4659.795306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4659.869589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4668.567019] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4673.190703] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4675.046436] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:17:44 (1787865464) [ 4945.891486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4945.894419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.913077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.913485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.917456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.930592] Lustre: Skipped 1 previous similar message [ 4946.955170] Lustre: Skipped 2 previous similar messages [ 4950.487711] Lustre: server umount lustre-MDT0000 complete [ 4952.044692] LustreError: 159438:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.073447] LustreError: 159438:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4954.985485] LustreError: 161400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865746 with bad export cookie 5681670696561291051 [ 4954.992407] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4955.012586] LustreError: 161400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4957.152119] LustreError: 163167:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4957.154224] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.193187] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4959.205575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4961.691823] Lustre: server umount lustre-MDT0001 complete [ 4972.504756] Lustre: server umount lustre-OST0000 complete [ 4984.753856] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4996.391624] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5036.826910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.062797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5037.085150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5037.159444] Lustre: lustre-MDT0000: new disk, initializing [ 5037.219663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5037.253499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5042.055668] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5052.826596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.979371] Lustre: 166067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5053.060397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5053.073090] Lustre: Skipped 1 previous similar message [ 5053.218419] Lustre: lustre-MDT0001: new disk, initializing [ 5053.347760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5053.391554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5053.400825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5057.985699] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5067.660801] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5073.725452] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5081.438095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5081.837912] Lustre: lustre-OST0000: new disk, initializing [ 5081.854880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5081.875776] Lustre: 168039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5081.971552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5088.787855] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5089.379770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5089.401713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5089.540410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5098.622654] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5103.556363] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5103.754692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5108.492796] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5108.763358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5113.316756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.336759] Lustre: Skipped 1 previous similar message [ 5113.341671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5113.360773] Lustre: Skipped 1 previous similar message [ 5116.867499] Lustre: server umount lustre-OST0000 complete [ 5123.553388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.557477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5123.571096] Lustre: Skipped 1 previous similar message [ 5126.904204] Lustre: server umount lustre-MDT0000 complete [ 5128.675767] LustreError: 166900:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.692907] LustreError: 166900:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5130.592378] LustreError: 166057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865922 with bad export cookie 5681670696561449874 [ 5130.598681] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5130.608914] LustreError: 166057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5130.864261] Lustre: server umount lustre-MDT0001 complete [ 5140.239519] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:25:30 (1787865930) [ 5157.446796] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 5166.660682] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5204.679155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5205.153989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5205.293548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5205.456476] Lustre: lustre-MDT0000: new disk, initializing [ 5205.606103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5205.630919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5210.824902] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5222.259960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.380672] Lustre: 172794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5222.764075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5227.550865] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5238.051509] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5243.371431] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5249.820244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5250.058824] Lustre: lustre-OST0000: new disk, initializing [ 5250.061405] Lustre: Skipped 1 previous similar message [ 5250.064055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5250.068141] Lustre: Skipped 2 previous similar messages [ 5250.071911] Lustre: 174769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5255.206435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5255.214183] Lustre: Skipped 1 previous similar message [ 5255.218843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5255.267777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5255.412636] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5264.062806] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5272.821952] Lustre: server umount lustre-OST0000 complete [ 5279.201970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5279.228466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.240863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.243645] Lustre: Skipped 1 previous similar message [ 5287.113178] Lustre: server umount lustre-MDT0000 complete [ 5290.697720] LustreError: 173594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866082 with bad export cookie 5681670696561450763 [ 5290.701887] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.708932] LustreError: 173594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5300.428670] Lustre: server umount lustre-MDT0001 complete [ 5309.494776] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:28:19 (1787866099) [ 5324.409499] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 5333.563126] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5368.189510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.430664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5368.474572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5368.615326] Lustre: lustre-MDT0000: new disk, initializing [ 5368.733611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5368.739927] Lustre: Skipped 2 previous similar messages [ 5368.754179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5373.166824] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5384.232912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.378332] Lustre: 179091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5384.447128] LustreError: 179104:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5384.653633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5388.910222] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5398.362904] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5404.131842] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5410.551522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5411.000753] Lustre: 181069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5417.139686] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5418.054610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5418.156250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5427.088964] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5438.951119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.954775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.961498] LustreError: Skipped 1 previous similar message [ 5438.964727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.964734] Lustre: Skipped 4 previous similar messages [ 5438.990793] Lustre: Skipped 2 previous similar messages [ 5441.352799] Lustre: server umount lustre-OST0000 complete [ 5446.113021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5451.653508] Lustre: server umount lustre-MDT0000 complete [ 5454.308220] LustreError: 181366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.322340] LustreError: 181366:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5455.885964] LustreError: 179906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866247 with bad export cookie 5681670696561451848 [ 5455.893534] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.898119] LustreError: 179906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5456.294576] Lustre: server umount lustre-MDT0001 complete [ 5468.102677] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5470.851321] Key type lgssc unregistered [ 5471.153871] LNet: 182932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5471.163287] LNetError: 182932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5471.178323] LNet: Removed LNI 192.168.203.116@tcp [ 5472.184807] Key type .llcrypt unregistered [ 5472.189757] Key type ._llcrypt unregistered [ 5497.530646] Key type ._llcrypt registered [ 5497.534242] Key type .llcrypt registered [ 5497.655313] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5499.274410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5499.361124] alg: No test for adler32 (adler32-zlib) [ 5500.598719] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 5500.831277] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5502.519161] Key type lgssc registered [ 5503.959441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5512.344143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5512.361705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.729172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.622953] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5523.941297] LustreError: 184351:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5523.971115] LustreError: 184351:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5528.188995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5528.812563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5533.708754] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5541.418540] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5547.054845] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5554.232735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5554.678290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5558.775282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5561.218325] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5569.428481] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5594.595838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5594.596017] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.605839] LustreError: Skipped 1 previous similar message [ 5594.609675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5594.627848] Lustre: Skipped 1 previous similar message [ 5594.646139] Lustre: Skipped 1 previous similar message [ 5596.363711] Lustre: server umount lustre-OST0000 complete [ 5600.235148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5600.254314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.274674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5604.836050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.837444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5604.866974] Lustre: Skipped 1 previous similar message [ 5606.693899] Lustre: server umount lustre-MDT0000 complete [ 5609.955413] LustreError: 184351:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5609.980284] LustreError: 184351:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5610.072221] LustreError: 184332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866401 with bad export cookie 2653266135092882369 [ 5610.077923] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5610.087605] LustreError: 184332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5610.392689] Lustre: server umount lustre-MDT0001 complete [ 5619.989345] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5622.605593] Key type lgssc unregistered [ 5622.872567] LNet: 187911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5622.889092] LNetError: 187911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5622.920003] LNet: Removed LNI 192.168.203.116@tcp [ 5623.703645] Key type .llcrypt unregistered [ 5623.705596] Key type ._llcrypt unregistered [ 5647.426615] Key type ._llcrypt registered [ 5647.428519] Key type .llcrypt registered [ 5647.606704] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5648.934382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5648.960848] alg: No test for adler32 (adler32-zlib) [ 5650.015722] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 5650.303645] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5652.015649] Key type lgssc registered [ 5653.161865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5660.997544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5661.021864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.351439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5666.525660] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5672.935307] LustreError: 189327:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5672.948210] LustreError: 189327:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5675.189119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.518897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5680.054806] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5687.533862] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5693.399847] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5700.775785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5701.035785] LustreError: 191145:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.172267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.278711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5708.123987] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5716.679697] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5724.801575] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 5726.689927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5726.699820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.719545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5727.720438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.735463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5727.740553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5727.746361] Lustre: Skipped 1 previous similar message [ 5729.938466] Lustre: server umount lustre-OST0000 complete [ 5737.739554] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f8746f81800.ost_server_uuid 50 [ 5739.478185] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5757.409824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5757.421988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.429433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5757.445585] Lustre: Skipped 1 previous similar message [ 5757.463463] Lustre: Skipped 1 previous similar message [ 5759.963557] Lustre: server umount lustre-MDT0000 complete [ 5762.531770] LustreError: 189323:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5762.568074] LustreError: 189323:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5763.286275] LustreError: 189308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866555 with bad export cookie 10432107838398565699 [ 5763.289495] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.298364] LustreError: 189308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5763.680985] Lustre: server umount lustre-MDT0001 complete [ 5772.466618] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:36:02 (1787866562) [ 5780.120359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.597778] LustreError: 193244:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5780.693294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.738822] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5786.081181] LustreError: 193245:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.297542] LustreError: 193244:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.333256] LustreError: 193244:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5795.442973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5796.013396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5801.063247] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5809.588521] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5815.641366] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5822.671334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.069565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5826.166230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5830.208493] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5838.580754] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5845.087434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5845.290601] Lustre: lustre-OST0001: new disk, initializing [ 5845.295887] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5845.308948] Lustre: 196086:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5845.428770] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5851.530311] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5852.810873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5852.817277] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5852.949688] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5860.265643] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5863.408477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.413636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5863.437930] Lustre: Skipped 1 previous similar message [ 5863.454266] Lustre: Skipped 1 previous similar message [ 5868.599172] Lustre: server umount lustre-OST0000 complete [ 5871.214099] LustreError: 195067:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.115864] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5876.337683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5880.935512] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5881.163502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5894.115740] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5894.117801] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.130079] LustreError: Skipped 1 previous similar message [ 5894.133353] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5894.139286] Lustre: Skipped 1 previous similar message [ 5894.156437] Lustre: Skipped 4 previous similar messages [ 5900.262683] Lustre: server umount lustre-OST0001 complete [ 5904.357395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.367935] Lustre: Skipped 1 previous similar message [ 5909.997183] Lustre: server umount lustre-MDT0000 complete [ 5914.433930] LustreError: 193225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866706 with bad export cookie 10432107838398566770 [ 5914.449994] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5914.463995] LustreError: 193225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5915.189224] Lustre: server umount lustre-MDT0001 complete [ 5943.173241] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:38:52 (1787866732) [ 5950.139121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.453678] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5950.524566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5950.807856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5954.862738] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5963.085912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.230343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5963.256347] Lustre: 199527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5968.055620] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5975.051788] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5980.869631] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5988.145641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.408209] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5994.917240] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5995.610916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6002.416838] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6009.104871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6009.238840] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6016.215078] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6024.405853] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6027.754227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6027.756464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6027.771722] Lustre: Skipped 1 previous similar message [ 6027.784254] Lustre: Skipped 7 previous similar messages [ 6032.991852] Lustre: server umount lustre-OST0000 complete [ 6035.571045] LustreError: 203032:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.593754] LustreError: 203032:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 16 previous similar messages [ 6053.344920] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6053.363460] LustreError: Skipped 1 previous similar message [ 6055.161567] Lustre: server umount lustre-OST0001 complete [ 6061.024359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.042107] Lustre: Skipped 2 previous similar messages [ 6061.066658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6061.069552] Lustre: Skipped 5 previous similar messages [ 6073.312996] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6073.811843] Lustre: server umount lustre-MDT0000 complete [ 6077.891401] LustreError: 199519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866869 with bad export cookie 10432107838398568177 [ 6077.906519] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6078.395054] Lustre: server umount lustre-MDT0001 complete [ 6102.166056] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:41:32 (1787866892) [ 6118.017104] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6127.296619] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6163.883124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.158963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6164.189566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6164.316550] Lustre: lustre-MDT0000: new disk, initializing [ 6164.385959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.390912] Lustre: Skipped 3 previous similar messages [ 6164.401878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6169.081788] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6179.187361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6179.262484] Lustre: 207620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6179.301108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6179.304273] Lustre: Skipped 1 previous similar message [ 6179.349144] Lustre: lustre-MDT0001: new disk, initializing [ 6179.440235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6179.450189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6183.414416] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6193.918520] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6200.538456] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6208.501213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6208.788359] Lustre: lustre-OST0000: new disk, initializing [ 6208.791266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6208.796620] Lustre: 209591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6216.224197] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6216.785806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6216.792446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6216.944283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6225.841846] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6229.513872] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6229.667238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6233.261835] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6233.426286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6237.673234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.673690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6237.681441] Lustre: Skipped 1 previous similar message [ 6237.685924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6237.691345] LustreError: Skipped 1 previous similar message [ 6237.716168] Lustre: Skipped 5 previous similar messages [ 6241.091370] Lustre: server umount lustre-OST0000 complete [ 6250.973952] Lustre: server umount lustre-MDT0000 complete [ 6253.040621] LustreError: 207626:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6253.067468] LustreError: 207626:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 6254.611895] LustreError: 208426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867046 with bad export cookie 10432107838398569731 [ 6254.613610] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6254.624217] LustreError: 208426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6254.955091] Lustre: server umount lustre-MDT0001 complete [ 6264.814518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6270.530027] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6279.313041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6284.848273] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6292.537207] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6298.916795] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6306.198753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6306.645278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6306.658943] Lustre: Skipped 4 previous similar messages [ 6313.514824] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6321.685132] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6327.202329] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6327.537444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6332.535037] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6332.781203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6335.463592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6335.473386] LustreError: Skipped 2 previous similar messages [ 6349.279182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6349.624803] Lustre: server umount lustre-OST0000 complete [ 6355.994788] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6360.335954] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6389.352433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6396.314338] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6401.001298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.016301] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.017774] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6401.047385] Lustre: Skipped 1 previous similar message [ 6403.623061] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6407.216832] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6411.246417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6411.246823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6411.249624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6411.249633] Lustre: Skipped 9 previous similar messages [ 6411.254759] LustreError: Skipped 1 previous similar message [ 6411.292583] Lustre: Skipped 6 previous similar messages [ 6417.340947] Lustre: server umount lustre-OST0000 complete [ 6427.553438] Lustre: server umount lustre-MDT0000 complete [ 6430.680987] LustreError: 211790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867222 with bad export cookie 10432107838398570620 [ 6430.684089] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6430.700380] LustreError: 211790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6431.128940] Lustre: server umount lustre-MDT0001 complete [ 6439.687340] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:47:09 (1787867229) [ 6446.344848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.920596] LustreError: 217776:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.948179] LustreError: 217776:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6451.334617] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6459.948237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.739967] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6473.861513] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6479.896260] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6487.012737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6494.024500] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6501.543504] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6506.215296] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6506.397451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6510.941557] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6511.250137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6517.592151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6517.737435] Lustre: lustre-OST0001: new disk, initializing [ 6517.740739] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6517.746342] Lustre: 221146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6523.960643] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6527.586951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6527.611982] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6527.700743] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6533.764919] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6538.476785] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6538.720544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6543.201199] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6543.439354] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6548.460253] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6548.471634] LustreError: Skipped 1 previous similar message [ 6551.416977] Lustre: server umount lustre-OST0001 complete [ 6557.225687] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6557.407989] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6561.664100] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6561.899666] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6591.222765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6591.382054] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6591.389871] Lustre: Skipped 5 previous similar messages [ 6596.597116] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6596.616691] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6596.618701] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6598.577678] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6606.528931] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6608.232851] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6624.223908] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6624.295766] Lustre: server umount lustre-OST0001 complete [ 6647.950926] Lustre: server umount lustre-OST0000 complete [ 6667.231210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6667.747533] Lustre: server umount lustre-MDT0000 complete [ 6672.031153] LustreError: 218512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867463 with bad export cookie 10432107838398571586 [ 6672.032687] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6672.043526] LustreError: 218512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6678.653070] Lustre: server umount lustre-MDT0001 complete [ 6702.417413] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:51:32 (1787867492) [ 6710.625402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.034849] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6711.110518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6716.772118] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6725.865050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.013856] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6726.044248] Lustre: 226395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6731.000557] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6739.077717] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6744.442355] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6751.886029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.075163] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6758.549522] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6766.253407] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6774.863611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6774.963464] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6781.057626] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6789.109379] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6790.658850] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6795.100703] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6795.339756] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6799.594117] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6799.823809] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6805.780733] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f87434df800.ost_server_uuid 50 [ 6807.529902] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f87434df800.ost_server_uuid in IDLE state after 0 sec [ 6808.481185] Lustre: Permanently deactivating lustre-OST0001 [ 6808.503305] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6812.049896] Lustre: Permanently reactivating lustre-OST0001 [ 6819.305960] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.332470] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6819.337341] Lustre: Skipped 11 previous similar messages [ 6819.361100] Lustre: Skipped 24 previous similar messages [ 6819.586738] Lustre: server umount lustre-OST0001 complete [ 6824.426542] LustreError: 228558:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6824.464468] LustreError: 228558:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 40 previous similar messages [ 6831.656666] Lustre: server umount lustre-OST0000 complete [ 6838.752127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6838.758526] LustreError: Skipped 2 previous similar messages [ 6844.132912] Lustre: server umount lustre-MDT0000 complete [ 6848.490443] LustreError: 229268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867640 with bad export cookie 10432107838398572874 [ 6848.493152] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6848.503384] LustreError: 229268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6848.976871] Lustre: server umount lustre-MDT0001 complete [ 6873.288703] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:54:23 (1787867663) [ 6881.698715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.070197] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6882.104403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6882.111725] Lustre: Skipped 5 previous similar messages [ 6886.707560] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6895.641183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.773657] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6895.837550] Lustre: 233095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6900.842634] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6907.796390] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6913.778445] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6921.062804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6921.400581] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6921.437668] Lustre: Permanently deactivating lustre-OST0000 [ 6921.449639] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6928.958872] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6929.477560] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6937.515431] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6944.524308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.587895] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6951.283860] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6952.158919] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6958.899968] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6960.508303] Lustre: Permanently reactivating lustre-OST0000 [ 6960.521305] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6960.525515] Lustre: Skipped 2 previous similar messages [ 6960.597308] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6960.631371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6967.341522] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6967.347762] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6967.367626] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6967.370691] Lustre: Skipped 1 previous similar message [ 6979.289572] Lustre: server umount lustre-OST0001 complete [ 6989.603602] Lustre: server umount lustre-OST0000 complete [ 7000.988679] Lustre: server umount lustre-MDT0000 complete [ 7004.624940] LustreError: 233089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867796 with bad export cookie 10432107838398574554 [ 7004.638635] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7004.639218] LustreError: 233089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7005.158075] Lustre: server umount lustre-MDT0001 complete [ 7018.068199] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7020.983655] Key type lgssc unregistered [ 7021.270918] LNet: 237974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7021.286663] LNetError: 237974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7021.298828] LNet: Removed LNI 192.168.203.116@tcp [ 7022.248692] Key type .llcrypt unregistered [ 7022.252033] Key type ._llcrypt unregistered [ 7044.593623] Key type ._llcrypt registered [ 7044.595803] Key type .llcrypt registered [ 7044.751835] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7045.933832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7045.993860] alg: No test for adler32 (adler32-zlib) [ 7047.203676] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 7047.548844] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7049.311198] Key type lgssc registered [ 7050.895787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7061.612663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7061.666681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.181392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7067.787507] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7072.740783] LustreError: 239440:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.755451] LustreError: 239440:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7075.596327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7075.733387] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7075.767307] Lustre: 239429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7075.780710] Lustre: Permanently deactivating lustre-MDT0001 [ 7075.809531] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7076.308824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7078.800453] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7078.808312] LustreError: 239471:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7078.825515] LustreError: 239471:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7078.840608] LustreError: 239471:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7081.301441] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7089.667039] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7095.390608] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7102.393128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7102.854073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7108.010272] LustreError: 241283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7108.027970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7109.144678] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7113.193616] LustreError: 241282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7116.969721] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7118.309064] LustreError: 241284:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.249280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.460237] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7129.608987] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7129.612881] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7130.243761] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7138.309899] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7142.373401] Lustre: Permanently reactivating lustre-MDT0001 [ 7142.415763] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7142.421178] Lustre: Skipped 2 previous similar messages [ 7147.702144] Lustre: lustre-MDT0001: Client f970624a-1bf2-49c5-a501-957bf203b166 (at 192.168.203.16@tcp) reconnecting [ 7153.543874] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7155.163472] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7161.875236] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7162.142599] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7166.805462] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7167.043718] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7170.257153] Lustre: Permanently deactivating lustre-MDT0001 [ 7170.279660] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7170.289569] Lustre: Skipped 2 previous similar messages [ 7179.406694] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7188.960559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7188.964978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.989439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7191.010546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7191.015702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7191.020075] Lustre: Skipped 1 previous similar message [ 7191.042945] Lustre: Skipped 2 previous similar messages [ 7193.481162] Lustre: server umount lustre-MDT0000 complete [ 7196.140279] LustreError: 240185:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.171838] LustreError: 240185:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7197.737797] LustreError: 239420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867989 with bad export cookie 14839503109026762033 [ 7197.745819] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7197.757937] LustreError: 239420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7198.345339] Lustre: server umount lustre-MDT0001 complete [ 7205.085696] Lustre: server umount lustre-OST0000 complete [ 7217.548853] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:00:07 (1787868007) [ 7229.538329] Lustre: server umount lustre-OST0001 complete [ 7236.383962] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7246.580825] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7285.242791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.458203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7285.469231] Lustre: Skipped 2 previous similar messages [ 7285.489376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7285.557278] Lustre: lustre-MDT0000: new disk, initializing [ 7285.618954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7285.636822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7290.453672] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7301.925506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7302.036661] Lustre: 247732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7302.115891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7302.121055] Lustre: Skipped 1 previous similar message [ 7302.184808] Lustre: lustre-MDT0001: new disk, initializing [ 7302.256533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7302.323970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7302.331532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7306.576382] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7316.747965] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7325.193430] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7333.732116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7334.328346] Lustre: lustre-OST0000: new disk, initializing [ 7334.331981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7334.339357] Lustre: 249700:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7334.414491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7341.337086] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7344.285478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7344.296937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7344.408397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7350.615955] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7352.684403] LustreError: 247740:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7361.450948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.690497] Lustre: lustre-OST0001: new disk, initializing [ 7361.696544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7361.713575] Lustre: 250815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7362.762296] LustreError: 247740:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7362.799568] LustreError: 247738:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7368.355103] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7369.343388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7369.354903] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7369.505669] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7372.895123] LustreError: 247738:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7378.995744] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7380.770696] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7385.060077] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7385.060504] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.073419] LustreError: Skipped 1 previous similar message [ 7385.079141] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7385.086556] Lustre: Skipped 1 previous similar message [ 7385.093478] Lustre: Skipped 2 previous similar messages [ 7388.843424] Lustre: server umount lustre-OST0001 complete [ 7390.190555] LustreError: 249683:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.207089] LustreError: 249683:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7410.659572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7410.661317] LustreError: 250165:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7410.667229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7410.667237] Lustre: Skipped 1 previous similar message [ 7410.678654] Lustre: Skipped 1 previous similar message [ 7410.704324] LustreError: 250165:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 7413.774444] Lustre: server umount lustre-OST0000 complete [ 7419.872762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7419.880920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.899873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7419.905682] Lustre: Skipped 1 previous similar message [ 7424.499161] Lustre: server umount lustre-MDT0000 complete [ 7428.643661] LustreError: 247725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868220 with bad export cookie 14839503109026764630 [ 7428.650360] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.655257] LustreError: 247725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7435.163325] Lustre: server umount lustre-MDT0001 complete [ 7447.080193] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7449.836605] Key type lgssc unregistered [ 7450.125361] LNet: 252851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7450.155692] LNetError: 252851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7451.179457] LNet: Removed LNI 192.168.203.116@tcp [ 7452.062089] Key type .llcrypt unregistered [ 7452.064600] Key type ._llcrypt unregistered [ 7498.116728] Key type ._llcrypt registered [ 7498.118675] Key type .llcrypt registered [ 7498.290637] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7499.917903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7499.981596] alg: No test for adler32 (adler32-zlib) [ 7501.313450] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 7501.808144] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7503.650743] Key type lgssc registered [ 7504.803184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7512.849535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7512.882255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.130450] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7514.159354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7514.348439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.913220] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7528.370596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7528.468224] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7528.492652] Lustre: 255066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7528.762117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7533.012360] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7540.354102] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7546.317416] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7553.198486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7553.341142] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7553.512235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7559.869216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7560.152695] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7567.568450] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7619.562193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.567353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.603502] Lustre: Skipped 1 previous similar message [ 7624.690912] Lustre: server umount lustre-OST0000 complete [ 7628.808268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7638.919457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7639.132763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7640.177615] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7640.188606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7640.188782] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7640.211898] Lustre: Skipped 1 previous similar message [ 7645.796875] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7653.298309] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7696.867284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7696.881773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7696.885226] LustreError: Skipped 1 previous similar message [ 7696.916709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7696.916717] Lustre: Skipped 1 previous similar message [ 7698.848670] Lustre: server umount lustre-OST0000 complete [ 7703.011482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7703.028571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.046146] Lustre: Skipped 1 previous similar message [ 7703.069703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.083266] Lustre: Skipped 1 previous similar message [ 7707.108391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7707.109592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.049656] Lustre: server umount lustre-MDT0000 complete [ 7712.767307] LustreError: 255056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868504 with bad export cookie 3430290733640157911 [ 7712.778915] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7712.789660] LustreError: 255056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7713.137787] Lustre: server umount lustre-MDT0001 complete [ 7724.353594] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7726.610760] Key type lgssc unregistered [ 7726.850902] LNet: 260486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7726.859184] LNetError: 260486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7726.876685] LNet: Removed LNI 192.168.203.116@tcp [ 7727.816420] Key type .llcrypt unregistered [ 7727.819522] Key type ._llcrypt unregistered [ 7757.873775] Key type ._llcrypt registered [ 7757.875663] Key type .llcrypt registered [ 7757.942561] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7758.418352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7758.426312] alg: No test for adler32 (adler32-zlib) [ 7759.574520] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 7759.787435] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7761.424130] Key type lgssc registered [ 7762.398731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7770.017823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7770.034543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.520487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7775.618831] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7781.862931] LustreError: 261903:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7781.894210] LustreError: 261903:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7784.551448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.996181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7789.440247] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7797.216704] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7802.862654] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7809.431943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.785652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.051465] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7816.945681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7823.686580] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7846.880487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7846.887495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.911389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7847.904181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7847.911139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7847.929828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7847.940649] Lustre: Skipped 1 previous similar message [ 7852.835369] Lustre: server umount lustre-OST0000 complete [ 7858.145184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.146792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7858.169351] Lustre: Skipped 1 previous similar message [ 7858.188094] Lustre: Skipped 1 previous similar message [ 7862.978488] Lustre: server umount lustre-MDT0000 complete [ 7863.266606] LustreError: 261904:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7863.295517] LustreError: 261904:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7866.558368] LustreError: 261885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868658 with bad export cookie 7108055558547566630 [ 7866.565147] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7866.578135] LustreError: 261885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7866.989422] Lustre: server umount lustre-MDT0001 complete [ 7877.451161] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7880.190684] Key type lgssc unregistered [ 7880.471981] LNet: 266215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7880.482795] LNetError: 266215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7880.510548] LNet: Removed LNI 192.168.203.116@tcp [ 7881.272721] Key type .llcrypt unregistered [ 7881.274640] Key type ._llcrypt unregistered [ 7897.323664] Key type ._llcrypt registered [ 7897.328838] Key type .llcrypt registered [ 7897.436976] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7898.596657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7898.614134] alg: No test for adler32 (adler32-zlib) [ 7899.673603] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 7899.931827] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7901.551161] Key type lgssc registered [ 7902.591692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7911.410677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7911.449489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7913.100271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7918.585086] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7922.658718] LustreError: 267438:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7922.668411] LustreError: 267438:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7926.958077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.282421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7930.998928] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7938.929201] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7944.569760] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7951.548140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7951.979294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7958.005804] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7958.133468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7964.868671] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7978.979133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7978.980546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.989900] LustreError: Skipped 1 previous similar message [ 7979.004975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7979.018720] Lustre: Skipped 1 previous similar message [ 7983.517659] Lustre: server umount lustre-OST0000 complete [ 7988.705091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7988.711395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7988.725058] Lustre: Skipped 1 previous similar message [ 7988.734629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7993.309914] Lustre: server umount lustre-MDT0000 complete [ 7994.337733] LustreError: 268166:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7994.361566] LustreError: 268166:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7996.836952] LustreError: 267419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868788 with bad export cookie 16073879775892756299 [ 7996.842417] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7996.847097] LustreError: 267419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7997.104895] Lustre: server umount lustre-MDT0001 complete [ 8007.681604] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8010.696186] Key type lgssc unregistered [ 8011.013908] LNet: 271120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8011.023915] LNetError: 271120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8011.041369] LNet: Removed LNI 192.168.203.116@tcp [ 8011.961869] Key type .llcrypt unregistered [ 8011.965809] Key type ._llcrypt unregistered [ 8037.139931] Key type ._llcrypt registered [ 8037.146395] Key type .llcrypt registered [ 8037.245985] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8038.453300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8038.477521] alg: No test for adler32 (adler32-zlib) [ 8039.510286] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 8039.765797] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8041.431158] Key type lgssc registered [ 8042.512456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8049.468502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8049.484090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.982422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8055.340276] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8060.384696] LustreError: 272548:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8060.421768] LustreError: 272548:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8063.792461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8064.133754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8068.805666] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8077.584443] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8083.697333] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8090.988676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8091.314663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8095.423737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8097.860527] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8106.014329] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8131.552798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8131.558959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.594133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8134.624704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8134.628511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8134.648147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8136.375338] Lustre: server umount lustre-OST0000 complete [ 8139.746360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.754767] Lustre: Skipped 1 previous similar message [ 8139.756055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8139.766573] Lustre: Skipped 2 previous similar messages [ 8144.867276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8144.878444] Lustre: Skipped 1 previous similar message [ 8145.696229] Lustre: server umount lustre-MDT0000 complete [ 8149.458984] LustreError: 272529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868941 with bad export cookie 16511536088792180081 [ 8149.474561] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8149.483972] LustreError: 272529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8149.964853] Lustre: server umount lustre-MDT0001 complete [ 8162.012623] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8165.131929] Key type lgssc unregistered [ 8165.435034] LNet: 276859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8165.442756] LNetError: 276859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8165.459430] LNet: Removed LNI 192.168.203.116@tcp [ 8166.382972] Key type .llcrypt unregistered [ 8166.387187] Key type ._llcrypt unregistered [ 8183.163525] Key type ._llcrypt registered [ 8183.165054] Key type .llcrypt registered [ 8183.283500] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8184.211388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8184.251954] alg: No test for adler32 (adler32-zlib) [ 8185.294642] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 8185.545758] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8187.175176] Key type lgssc registered [ 8188.462621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8196.427764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8196.459102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8198.120672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8203.788409] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8207.330962] LustreError: 278096:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.340046] LustreError: 278096:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8212.482996] LustreError: 278098:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8212.851799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8213.181571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8219.349688] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8228.316364] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8234.890856] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8242.126972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8242.477375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8249.079706] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8256.384647] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8268.075373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8278.499783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8278.500899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.524583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8278.545484] Lustre: Skipped 1 previous similar message [ 8278.553013] Lustre: Skipped 1 previous similar message [ 8283.205628] Lustre: server umount lustre-OST0000 complete [ 8288.744726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8288.746778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8288.755602] Lustre: Skipped 1 previous similar message [ 8288.773463] Lustre: Skipped 1 previous similar message [ 8292.788077] Lustre: server umount lustre-MDT0000 complete [ 8293.855912] LustreError: 278109:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8293.872425] LustreError: 278109:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8296.003889] LustreError: 281065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869087 with bad export cookie 13074315777345141260 [ 8296.004914] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8296.019561] LustreError: 281065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8296.328603] Lustre: server umount lustre-MDT0001 complete [ 8306.294493] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8309.207693] Key type lgssc unregistered [ 8309.555607] LNet: 281820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8309.561544] LNetError: 281820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8309.593396] LNet: Removed LNI 192.168.203.116@tcp [ 8310.580592] Key type .llcrypt unregistered [ 8310.583727] Key type ._llcrypt unregistered [ 8321.936314] Key type ._llcrypt registered [ 8321.943320] Key type .llcrypt registered [ 8322.072870] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8338.089340] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8351.659615] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8352.724970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8352.765989] alg: No test for adler32 (adler32-zlib) [ 8353.854867] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 8354.122476] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8355.775165] Key type lgssc registered [ 8356.840750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8394.037969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8394.099113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.329676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8395.365866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8395.471608] Lustre: lustre-MDT0000: new disk, initializing [ 8395.553092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.580105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8400.199703] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8412.413571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.597378] Lustre: 285375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8412.759503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8412.764393] Lustre: Skipped 1 previous similar message [ 8412.867380] Lustre: lustre-MDT0001: new disk, initializing [ 8413.001826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8413.039454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8413.056432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8418.750083] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8429.061989] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8435.874626] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8443.204573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8443.486299] Lustre: lustre-OST0000: new disk, initializing [ 8443.493435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8443.502108] Lustre: 287344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8443.589962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8450.350295] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8451.159485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8451.167272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8451.304854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8461.103723] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8466.258534] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8466.513888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8471.401211] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8471.651673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8477.155164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8477.156981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8477.170680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8477.205649] Lustre: Skipped 1 previous similar message [ 8480.050231] Lustre: server umount lustre-OST0000 complete [ 8484.832850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8484.843104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8484.856407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8484.863898] Lustre: Skipped 1 previous similar message [ 8487.393197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.399032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8487.417416] Lustre: Skipped 1 previous similar message [ 8490.384565] Lustre: server umount lustre-MDT0000 complete [ 8494.726581] LustreError: 287343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869286 with bad export cookie 6375997171898620691 [ 8494.729144] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8494.737523] LustreError: 287343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8495.152914] Lustre: server umount lustre-MDT0001 complete [ 8505.714295] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:21:35 (1787869295) [ 8513.060204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.672060] LustreError: 289755:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8513.694568] LustreError: 289755:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8513.746509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8518.591499] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8519.138368] LustreError: 289756:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8523.238851] LustreError: 289755:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.579640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.953085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8532.617954] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8540.194381] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8546.645638] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8553.483546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8553.851514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8559.685153] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8567.109683] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8582.625953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8582.632773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8582.634671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8582.661309] Lustre: Skipped 1 previous similar message [ 8587.072820] Lustre: server umount lustre-OST0000 complete [ 8592.867835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.870646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8592.876372] Lustre: Skipped 1 previous similar message [ 8592.884627] Lustre: Skipped 2 previous similar messages [ 8603.109414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8603.120297] Lustre: Skipped 3 previous similar messages [ 8605.151238] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8605.430460] Lustre: server umount lustre-MDT0000 complete [ 8608.231153] LustreError: 292245:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8608.257914] LustreError: 292245:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8609.735342] LustreError: 289736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869401 with bad export cookie 6375997171898621573 [ 8609.756336] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8609.767515] LustreError: 289736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8610.078408] Lustre: server umount lustre-MDT0001 complete [ 8621.511042] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8624.275904] Key type lgssc unregistered [ 8624.606256] LNet: 293316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8624.622730] LNetError: 293316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8624.644626] LNet: Removed LNI 192.168.203.116@tcp [ 8625.480850] Key type .llcrypt unregistered [ 8625.488317] Key type ._llcrypt unregistered [ 8652.350559] Key type ._llcrypt registered [ 8652.353214] Key type .llcrypt registered [ 8652.465591] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8666.351737] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8667.509765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8667.560648] alg: No test for adler32 (adler32-zlib) [ 8668.645313] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 8668.941146] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8670.631231] Key type lgssc registered [ 8671.622114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8715.833168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8715.870600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.096835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8717.139612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8717.223376] Lustre: lustre-MDT0000: new disk, initializing [ 8717.277880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8717.290330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8721.352286] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8730.997619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.150215] Lustre: 297138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8731.191320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8731.194724] Lustre: Skipped 1 previous similar message [ 8731.269934] Lustre: lustre-MDT0001: new disk, initializing [ 8731.369345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8731.422127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8731.432948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8736.274516] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8745.359487] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8750.444591] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8757.295604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8757.564374] Lustre: lustre-OST2710: new disk, initializing [ 8757.573625] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8757.583580] Lustre: 299109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8757.650870] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8762.939923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8762.963975] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8763.038666] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8764.934481] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8774.277141] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8780.534830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8780.706586] Lustre: lustre-OST03e8: new disk, initializing [ 8780.712525] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8780.719995] Lustre: 300172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8780.823569] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8787.079232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8787.088821] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8787.173773] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8787.515081] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8796.206761] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8803.126951] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8803.370406] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8808.472990] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8808.753718] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8813.599081] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8813.807423] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8818.719687] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8818.970170] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8823.776866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8823.793445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8823.809056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8826.002942] Lustre: server umount lustre-MDT0000 complete [ 8828.396280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.411639] Lustre: Skipped 2 previous similar messages [ 8829.433826] LustreError: 297130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869621 with bad export cookie 16803930687318083897 [ 8829.435026] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8829.451675] LustreError: 297130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8830.074877] Lustre: server umount lustre-MDT0001 complete [ 8844.334632] Lustre: server umount lustre-OST2710 complete [ 8857.417186] Lustre: server umount lustre-OST03e8 complete [ 8877.149700] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8887.070318] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8921.885494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.119263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8922.149868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8922.215530] Lustre: lustre-MDT0000: new disk, initializing [ 8922.314970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8922.326539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8926.584790] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8937.018058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.164403] Lustre: 305421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8937.235178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8937.241056] Lustre: Skipped 1 previous similar message [ 8937.388535] Lustre: lustre-MDT0001: new disk, initializing [ 8937.470015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8937.502552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8937.516593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8941.977579] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8951.042512] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8957.618962] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8964.549953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8964.860152] Lustre: lustre-OST0000: new disk, initializing [ 8964.864420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8964.876207] Lustre: 307389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8964.962614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8971.336159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8971.359573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8971.430714] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8971.484188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8981.992626] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8986.744509] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8986.966198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8991.470674] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8991.694513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8997.347342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8997.353211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8997.368342] Lustre: Skipped 1 previous similar message [ 8999.783051] Lustre: server umount lustre-OST0000 complete [ 9004.002064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9004.017350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9004.035238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9004.042362] Lustre: Skipped 1 previous similar message [ 9007.585979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9012.705396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9012.709286] Lustre: Skipped 2 previous similar messages [ 9018.335109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9018.627308] Lustre: server umount lustre-MDT0000 complete [ 9022.218736] LustreError: 305413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869814 with bad export cookie 16803930687318085577 [ 9022.220139] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9022.228437] LustreError: 305413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9022.590614] Lustre: server umount lustre-MDT0001 complete [ 9031.415336] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:30:21 (1787869821) [ 9033.035880] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9035.193811] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:30:24 (1787869824) [ 9038.705320] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9065.419768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9065.679743] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9065.722387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9065.938218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9070.927935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9083.466069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9083.767892] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9083.855667] Lustre: 311248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9092.818733] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9115.888289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9116.126476] Lustre: Denying initial registration attempt from nid 192.168.203.116@tcp, specified as failover [ 9116.140868] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9116.153543] LustreError: 312702:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9116.163075] LustreError: 312702:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9116.170628] LustreError: 312702:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9116.273483] Lustre: server umount lustre-OST0000 complete [ 9116.275464] LustreError: 312702:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9125.344554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9125.355817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.373031] Lustre: Skipped 1 previous similar message [ 9125.387526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9125.393550] Lustre: Skipped 2 previous similar messages [ 9129.436855] Lustre: server umount lustre-MDT0000 complete [ 9132.513183] LustreError: 311260:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.521076] LustreError: 311260:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 9133.228121] LustreError: 311242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869925 with bad export cookie 16803930687318086459 [ 9133.230702] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9133.232857] LustreError: 311242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9133.485385] Lustre: server umount lustre-MDT0001 complete [ 9145.228470] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9149.747931] Key type lgssc unregistered [ 9150.330420] LNet: 313827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9150.349165] LNetError: 313827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9150.376541] LNet: Removed LNI 192.168.203.116@tcp [ 9151.084102] Key type .llcrypt unregistered [ 9151.088866] Key type ._llcrypt unregistered [ 9165.448864] Key type ._llcrypt registered [ 9165.452535] Key type .llcrypt registered [ 9165.588669] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9167.057414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9167.094723] alg: No test for adler32 (adler32-zlib) [ 9168.194095] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 9168.472250] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9170.191834] Key type lgssc registered [ 9171.279868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9217.581205] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9226.248564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9226.283798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9227.572438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9227.594578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9227.837809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9232.718488] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9241.205222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9241.347681] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9241.404564] Lustre: 316592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9241.764846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9246.768791] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9254.569945] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9260.723237] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9281.327392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9281.602511] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9281.912317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.062792] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9296.707305] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9303.524877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9303.548783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9303.589662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9304.035400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9305.709680] Lustre: server umount lustre-OST0000 complete [ 9313.250052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9313.266771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9313.286963] Lustre: Skipped 1 previous similar message [ 9313.304016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9313.324189] Lustre: Skipped 2 previous similar messages [ 9314.286136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.307537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9314.315536] Lustre: Skipped 1 previous similar message [ 9317.076501] Lustre: server umount lustre-MDT0000 complete [ 9321.002072] LustreError: 316583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870112 with bad export cookie 5733234975784440839 [ 9321.003690] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9321.024250] LustreError: 316583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9321.549738] Lustre: server umount lustre-MDT0001 complete [ 9333.909025] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9337.135231] Key type lgssc unregistered [ 9337.406855] LNet: 320170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9337.415441] LNetError: 320170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9337.443638] LNet: Removed LNI 192.168.203.116@tcp [ 9338.661551] Key type .llcrypt unregistered [ 9338.666089] Key type ._llcrypt unregistered [ 9365.086869] Key type ._llcrypt registered [ 9365.090719] Key type .llcrypt registered [ 9365.198764] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9366.346329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9366.358157] alg: No test for adler32 (adler32-zlib) [ 9367.369640] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 9367.635796] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9369.361515] Key type lgssc registered [ 9370.481930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9378.341540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9378.360705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.797937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.524832] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9390.057978] LustreError: 321589:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9390.080496] LustreError: 321589:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9393.418257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9393.872781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9399.378498] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9407.537958] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9413.901556] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9422.588751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9422.961201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9429.700437] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9438.232902] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9448.929748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9448.934634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9448.951356] Lustre: Skipped 2 previous similar messages [ 9448.958986] Lustre: Skipped 1 previous similar message [ 9451.182725] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 9451.198406] Lustre: Skipped 1 previous similar message [ 9453.976319] Lustre: server umount lustre-MDT0000 complete [ 9454.051571] LustreError: 322933:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9454.084387] LustreError: 322933:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9456.241725] LustreError: 321585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.117687] LustreError: 321570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870249 with bad export cookie 13881643934738686771 [ 9458.127926] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9458.145603] LustreError: 321570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9459.171849] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9459.175890] LustreError: 322933:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9459.190136] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9461.398849] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 9463.211524] LustreError: 322933:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9463.230994] LustreError: 322933:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9464.553585] Lustre: server umount lustre-MDT0001 complete [ 9471.248781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9480.394033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.745643] LustreError: 320898:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000370c78cf x1874717739882112/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9484.169238] LustreError: 323421:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9484.272652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9484.343836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9489.958350] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9493.485266] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9499.087362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9499.478568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9504.652873] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9504.762372] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9513.676717] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9515.178659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9520.946870] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9522.496923] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9530.336549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9530.347342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9530.369855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9530.375398] Lustre: Skipped 2 previous similar messages [ 9534.484748] Lustre: server umount lustre-OST0000 complete [ 9540.580104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9540.581789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9540.608603] Lustre: Skipped 2 previous similar messages [ 9540.623098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9540.641872] LustreError: Skipped 1 previous similar message [ 9546.163598] Lustre: server umount lustre-MDT0000 complete [ 9550.827207] LustreError: 325186:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9550.863055] LustreError: 325186:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 9551.653699] LustreError: 321568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870343 with bad export cookie 13881643934738694303 [ 9551.658442] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9551.672692] LustreError: 321568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9551.948075] Lustre: server umount lustre-MDT0001 complete [ 9565.352103] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9567.982569] Key type lgssc unregistered [ 9568.201810] LNet: 327860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9568.207105] LNetError: 327860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9568.226842] LNet: Removed LNI 192.168.203.116@tcp [ 9569.345113] Key type .llcrypt unregistered [ 9569.347858] Key type ._llcrypt unregistered [ 9595.940962] Key type ._llcrypt registered [ 9595.946255] Key type .llcrypt registered [ 9596.102519] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9597.444307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9597.475246] alg: No test for adler32 (adler32-zlib) [ 9598.674723] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 9599.014649] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9600.761528] Key type lgssc registered [ 9602.089656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9610.581912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9610.624899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9612.142482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9617.251052] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9621.476420] LustreError: 329281:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9621.496110] LustreError: 329281:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9626.593194] LustreError: 329282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9626.733537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9627.120832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9632.032504] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9641.442110] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9647.629279] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9651.683670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.709276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9651.712836] Lustre: Skipped 1 previous similar message [ 9651.729132] Lustre: Skipped 1 previous similar message [ 9652.770041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.776851] Lustre: Skipped 1 previous similar message [ 9657.826036] LustreError: 329276:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.859059] LustreError: 329276:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9659.408821] Lustre: server umount lustre-MDT0000 complete [ 9662.945413] LustreError: 330009:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9662.964092] LustreError: 330009:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9663.998996] LustreError: 330897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870455 with bad export cookie 5493009647819092867 [ 9664.002544] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9664.012115] LustreError: 330897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9664.468728] Lustre: server umount lustre-MDT0001 complete [ 9682.008483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9682.407471] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9682.446888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9682.670773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.386088] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9699.615772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9699.756722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9699.805163] Lustre: 331868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9700.346443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9705.470126] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9713.071019] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9718.835625] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9728.492601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9728.929424] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9728.941140] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9728.947300] LustreError: 333756:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9728.951554] LustreError: 333756:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9728.956377] LustreError: 333756:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9729.003522] Lustre: server umount lustre-OST0000 complete [ 9729.006252] LustreError: 333756:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9741.168604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9741.376851] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9741.504154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9748.872928] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9751.668456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9757.292639] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9766.491433] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9766.615201] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9766.674755] Lustre: lustre-OST0001: new disk, initializing [ 9766.678056] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9766.683930] Lustre: 335298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9766.781671] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9774.327908] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9775.753764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9775.759568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9775.906742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9782.803621] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9786.342771] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9786.346767] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.346870] LustreError: Skipped 1 previous similar message [ 9786.361325] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9786.372126] Lustre: Skipped 1 previous similar message [ 9791.471309] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9791.478743] Lustre: Skipped 1 previous similar message [ 9796.581872] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9796.591021] Lustre: Skipped 1 previous similar message [ 9800.160105] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9800.264975] Lustre: server umount lustre-OST0001 complete [ 9801.707779] LustreError: 334223:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9801.739236] LustreError: 334223:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9806.820410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.836345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9806.849878] Lustre: Skipped 2 previous similar messages [ 9806.878170] Lustre: Skipped 1 previous similar message [ 9810.799341] Lustre: server umount lustre-OST0000 complete [ 9817.063196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9817.083132] Lustre: Skipped 1 previous similar message [ 9821.770196] Lustre: server umount lustre-MDT0000 complete [ 9822.176540] LustreError: 331876:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9822.194915] LustreError: 331876:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9826.162498] LustreError: 331862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870618 with bad export cookie 5493009647819093420 [ 9826.171395] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9826.174866] LustreError: 331862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9826.623166] Lustre: server umount lustre-MDT0001 complete [ 9838.315902] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9841.662756] Key type lgssc unregistered [ 9842.105204] LNet: 337260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9842.121446] LNetError: 337260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9843.180949] LNet: Removed LNI 192.168.203.116@tcp [ 9844.231550] Key type .llcrypt unregistered [ 9844.236339] Key type ._llcrypt unregistered [ 9915.935715] Key type ._llcrypt registered [ 9915.937904] Key type .llcrypt registered [ 9916.044404] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9930.600892] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9931.865389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9931.944768] alg: No test for adler32 (adler32-zlib) [ 9933.262367] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [ 9933.600641] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9935.311166] Key type lgssc registered [ 9936.499570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9972.193240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9972.240200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.557645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9973.619350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9973.765613] Lustre: lustre-MDT0000: new disk, initializing [ 9973.851978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9973.871553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9978.943368] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9991.214150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9991.401524] Lustre: 342193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9991.519772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9991.523180] Lustre: Skipped 1 previous similar message [ 9991.667230] Lustre: lustre-MDT0001: new disk, initializing [ 9991.868296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9991.939369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9991.948054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9998.207652] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10009.340966] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10015.598440] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10022.437631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10022.743810] Lustre: lustre-OST0000: new disk, initializing [10022.753296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10022.765212] Lustre: 344165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10022.882800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10029.692361] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10032.691280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10032.709459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10032.820408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10040.107835] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10044.715991] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10044.924157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10049.936131] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10050.152791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10053.605646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.607991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10053.633491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10053.642433] Lustre: Skipped 1 previous similar message [10058.724496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10058.728884] Lustre: Skipped 1 previous similar message [10063.841743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10063.852256] Lustre: Skipped 1 previous similar message [10066.399294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10066.598921] Lustre: server umount lustre-OST0000 complete [10073.571241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10073.580097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10073.591710] Lustre: Skipped 1 previous similar message [10073.603140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10077.699400] Lustre: server umount lustre-MDT0000 complete [10081.716120] LustreError: 342185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870873 with bad export cookie 9585115299498964848 [10081.728249] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10081.728341] LustreError: 342185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10082.171145] Lustre: server umount lustre-MDT0001 complete [10090.374380] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:48:00 (1787870880) [10098.353502] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:48:08 (1787870888) [10108.488604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.049196] LustreError: 346923:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10109.075151] LustreError: 346923:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10109.141537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10114.529293] LustreError: 346924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10115.066539] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10118.639285] LustreError: 346924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.759572] LustreError: 346923:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.853570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10124.212569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10128.996751] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10137.038818] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10143.750515] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10150.416272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10150.759776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10156.962268] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10165.367791] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10178.436251] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10180.452224] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10182.154376] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10184.431609] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10186.610142] Lustre: Failing over lustre-MDT0000 [10186.941089] Lustre: server umount lustre-MDT0000 complete [10188.778085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10188.786920] LustreError: 346924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.798827] Lustre: Skipped 2 previous similar messages [10188.829967] LustreError: 346924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10199.013030] LustreError: 346923:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10199.039614] LustreError: 346923:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [10205.154414] Lustre: 340417:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870980/real 1787870980] req@0000000083cb297f x1874718333412352/t0(0) o400->MGC192.168.203.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787870996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10205.178472] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10206.735366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.392458] LustreError: 346924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10215.420669] LustreError: 346924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [10215.539807] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (not set up) [10215.562648] Lustre: Skipped 2 previous similar messages [10215.886298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10215.984403] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10220.657347] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10220.984398] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10221.034351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10221.062377] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10221.139316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10230.528749] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10232.144690] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10235.464466] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10237.355856] Lustre: Failing over lustre-MDT0000 [10237.685273] Lustre: server umount lustre-MDT0000 complete [10241.506240] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10241.515852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10241.531930] Lustre: Skipped 2 previous similar messages [10251.380833] LustreError: 346918:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10251.397231] LustreError: 346918:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [10256.687365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10256.894962] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10257.189732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10257.232754] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10257.589277] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10261.554148] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10262.512709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10262.518605] Lustre: Skipped 2 previous similar messages [10262.552421] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10262.600460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10271.098965] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10272.795323] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10282.978603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10282.985296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10282.992110] Lustre: Skipped 3 previous similar messages [10282.999511] Lustre: Skipped 2 previous similar messages [10284.838745] Lustre: server umount lustre-MDT0000 complete [10288.710419] LustreError: 346903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871080 with bad export cookie 9585115299498967557 [10288.716105] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10288.724422] LustreError: 346903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10289.155152] Lustre: server umount lustre-MDT0001 complete [10302.600921] Lustre: server umount lustre-OST0000 complete [10316.346726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10316.781253] LustreError: 353288:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10316.800869] LustreError: 353288:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [10321.332323] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10329.239187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10329.556247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10329.558950] Lustre: Skipped 1 previous similar message [10334.441258] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10342.768782] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10348.120870] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10354.301537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10360.832992] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10361.836554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10368.286472] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10376.162459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10376.177192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10376.206237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10379.346591] Lustre: server umount lustre-OST0000 complete [10385.891687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10385.898104] LustreError: Skipped 1 previous similar message [10389.280468] Lustre: server umount lustre-MDT0000 complete [10393.439815] LustreError: 353269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871185 with bad export cookie 9585115299498968306 [10393.457673] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10393.463457] LustreError: 353269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10393.790403] Lustre: server umount lustre-MDT0001 complete [10404.686656] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [10407.545230] Key type lgssc unregistered [10407.897139] LNet: 356823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10407.901162] LNetError: 356823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10407.931138] LNet: Removed LNI 192.168.203.116@tcp [10408.759818] Key type .llcrypt unregistered [10408.762846] Key type ._llcrypt unregistered [10437.802403] Key type ._llcrypt registered [10437.805320] Key type .llcrypt registered [10437.939389] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10439.715473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10439.816141] alg: No test for adler32 (adler32-zlib) [10440.864577] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [10441.160784] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [10442.863345] Key type lgssc registered [10444.271719] Lustre: Echo OBD driver; http://www.lustre.org/ [10452.207125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10452.238943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10453.663383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10458.357312] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10463.204499] LustreError: 358341:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10463.217023] LustreError: 358341:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10468.093919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10468.322176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10468.536588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10473.090854] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10480.187152] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10485.546464] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10491.525022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10491.930857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10497.164941] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10499.124424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10505.221606] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10518.857451] Lustre: server umount lustre-MDT0000 complete [10519.530924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10519.541456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10519.548622] LustreError: 358337:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10519.577441] LustreError: 358337:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10522.991383] LustreError: 360833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871314 with bad export cookie 6997444910062808445 [10522.997815] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10523.005345] LustreError: 360833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10523.413141] Lustre: server umount lustre-MDT0001 complete [10538.815461] Lustre: server umount lustre-OST0000 complete [10550.672224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.214804] LustreError: 362050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.284757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.737554] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10556.388917] LustreError: 362051:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.505771] LustreError: 362050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10564.001588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.409473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10568.782829] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10576.241735] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10582.291729] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10588.954914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10589.290125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10596.273172] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10596.465886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10604.147565] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10615.776117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10615.780871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.794111] Lustre: Skipped 2 previous similar messages [10615.800149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10617.314237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10617.317764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10617.328658] Lustre: Skipped 1 previous similar message [10617.349445] Lustre: Skipped 1 previous similar message [10622.097356] Lustre: server umount lustre-MDT0000 complete [10622.434525] LustreError: 362050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10622.454436] LustreError: 362050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10625.553048] LustreError: 362032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871417 with bad export cookie 6997444910062809565 [10625.563017] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10625.564713] LustreError: 362032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10625.827290] Lustre: server umount lustre-MDT0001 complete [10639.669326] Lustre: server umount lustre-OST0000 complete [10655.432231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10655.870932] LustreError: 365965:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10655.950826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10660.432631] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10669.166330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10673.382228] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10680.105548] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10685.726236] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10692.611074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10693.113847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10693.118591] Lustre: Skipped 1 previous similar message [10700.087162] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10700.278722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10708.495859] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10715.619046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10715.627887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10715.637516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10715.640155] Lustre: Skipped 1 previous similar message [10715.648866] Lustre: Skipped 1 previous similar message [10716.080421] Lustre: server umount lustre-OST0000 complete [10720.736707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10720.738411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10720.759206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10720.764285] LustreError: Skipped 1 previous similar message [10720.767028] Lustre: Skipped 2 previous similar messages [10726.134832] Lustre: server umount lustre-MDT0000 complete [10729.373762] LustreError: 365947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871521 with bad export cookie 6997444910062810587 [10729.375376] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10729.403309] LustreError: 365947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10729.823406] Lustre: server umount lustre-MDT0001 complete [10739.974927] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [10742.665425] Key type lgssc unregistered [10742.950831] LNet: 369499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10742.957714] LNetError: 369499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10742.982569] LNet: Removed LNI 192.168.203.116@tcp [10743.809741] Key type .llcrypt unregistered [10743.812171] Key type ._llcrypt unregistered [10771.436347] Key type ._llcrypt registered [10771.438449] Key type .llcrypt registered [10771.551645] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10772.320917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10772.409061] alg: No test for adler32 (adler32-zlib) [10773.812702] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [10774.326484] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [10776.143337] Key type lgssc registered [10777.818928] Lustre: Echo OBD driver; http://www.lustre.org/ [10786.768151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10786.786707] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10786.794145] LustreError: 370940:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10786.811474] LustreError: 370940:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10786.817199] LustreError: 370940:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10786.823392] LustreError: 370940:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10786.832946] LustreError: 370940:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10794.500932] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10794.509550] LustreError: 371337:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10794.537621] LustreError: 371337:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10794.550346] LustreError: 371337:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10794.560716] LustreError: 371337:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10794.570899] LustreError: 371337:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10814.531940] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [10817.553714] Key type lgssc unregistered [10817.833274] LNet: 372343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10817.843427] LNetError: 372343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10817.858890] LNet: Removed LNI 192.168.203.116@tcp [10818.633033] Key type .llcrypt unregistered [10818.635061] Key type ._llcrypt unregistered [10836.451272] Key type ._llcrypt registered [10836.455296] Key type .llcrypt registered [10836.634919] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [10853.900576] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10854.889581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10854.947208] alg: No test for adler32 (adler32-zlib) [10856.057968] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [10856.315200] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [10858.007178] Key type lgssc registered [10859.243180] Lustre: Echo OBD driver; http://www.lustre.org/ [10896.748091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10896.774725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.188844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10898.239786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10898.374684] Lustre: lustre-MDT0000: new disk, initializing [10898.450317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10898.475782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10903.257504] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10914.029987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.122996] Lustre: 375511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10914.177153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10914.181640] Lustre: Skipped 1 previous similar message [10914.240347] Lustre: lustre-MDT0001: new disk, initializing [10914.333125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10914.383884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10914.399154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10919.073376] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10927.728719] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10933.602194] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10942.209674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10942.686563] Lustre: lustre-OST0000: new disk, initializing [10942.689800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10942.699867] Lustre: 377481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10942.822234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10949.804072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10949.830112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10950.040037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10950.435384] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10963.006992] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10967.397777] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10967.553315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10972.240912] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10972.407984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10975.741241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10975.741907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10975.779924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10975.789609] Lustre: Skipped 1 previous similar message [10980.484454] Lustre: server umount lustre-OST0000 complete [10985.958306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10985.969189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10985.980883] LustreError: Skipped 1 previous similar message [10985.986093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10985.989654] Lustre: Skipped 2 previous similar messages [10990.999241] Lustre: server umount lustre-MDT0000 complete [10995.834274] LustreError: 375503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871787 with bad export cookie 13334547783636171276 [10995.851978] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10995.859757] LustreError: 375503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10996.293383] Lustre: server umount lustre-MDT0001 complete [11006.623370] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11008.179915] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:03:18 (1787871798) [11014.853965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11015.433231] LustreError: 379945:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11015.449802] LustreError: 379945:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11015.561128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11019.698337] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11020.774107] LustreError: 379946:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11025.897159] LustreError: 379945:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11028.537658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11028.857585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11033.951032] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11041.956668] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11047.465209] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11054.114628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11054.526252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11061.455820] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11069.607227] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11077.833860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11078.081483] Lustre: lustre-OST0001: new disk, initializing [11078.084918] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11078.092855] Lustre: 382779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11078.223994] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11083.894268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11083.905303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11084.005219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11085.961878] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11096.649473] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11103.389981] Lustre: lustre-OST0001: Not available for connect from 192.168.203.16@tcp (stopping) [11103.403976] Lustre: Skipped 1 previous similar message [11104.737339] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11104.750404] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.768409] Lustre: Skipped 1 previous similar message [11106.574841] Lustre: server umount lustre-OST0001 complete [11108.465687] LustreError: 381766:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11114.984293] LustreError: 381765:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11115.000706] LustreError: 381765:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11125.220122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11125.222898] LustreError: 383047:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.223821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11125.227727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11125.227734] Lustre: Skipped 2 previous similar messages [11125.231309] LustreError: Skipped 1 previous similar message [11125.286707] LustreError: 383047:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11129.947560] Lustre: server umount lustre-OST0000 complete [11135.456493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11135.476047] Lustre: Skipped 1 previous similar message [11135.492994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11135.499595] Lustre: Skipped 1 previous similar message [11140.531896] Lustre: server umount lustre-MDT0000 complete [11144.323677] LustreError: 383813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871936 with bad export cookie 13334547783636172158 [11144.330556] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11144.336017] LustreError: 383813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11150.856920] Lustre: server umount lustre-MDT0001 complete [11162.826150] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11165.914332] Key type lgssc unregistered [11166.228114] LNet: 384774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11166.232251] LNetError: 384774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11167.271620] LNet: Removed LNI 192.168.203.116@tcp [11168.344429] Key type .llcrypt unregistered [11168.346128] Key type ._llcrypt unregistered [11219.163484] Key type ._llcrypt registered [11219.165890] Key type .llcrypt registered [11219.286915] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11220.700647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11220.768170] alg: No test for adler32 (adler32-zlib) [11221.977559] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [11222.324821] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11224.055168] Key type lgssc registered [11225.400047] Lustre: Echo OBD driver; http://www.lustre.org/ [11234.218896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11234.257337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11235.561255] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11235.591032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11235.849049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11241.115694] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11251.015528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11251.143591] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11251.188093] Lustre: 387574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11251.359361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11251.537177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11256.371521] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11264.897728] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11271.193930] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11277.283118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11277.288909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11277.298197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11277.324992] Lustre: Skipped 1 previous similar message [11277.343085] Lustre: Skipped 1 previous similar message [11279.764087] Lustre: server umount lustre-MDT0000 complete [11284.605865] LustreError: 387567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872076 with bad export cookie 7142026299153505563 [11284.611730] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11284.634800] LustreError: 387567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11285.234710] Lustre: server umount lustre-MDT0001 complete [11297.202298] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:08:06 (1787872086) [11305.009157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11305.607883] LustreError: 390086:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.618629] LustreError: 390086:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11305.641669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11310.338203] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11311.091580] LustreError: 390086:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11316.208521] LustreError: 390087:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.326630] LustreError: 390086:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.892688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.543438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11327.500448] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11336.308639] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11342.627481] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11350.820225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11351.129697] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11351.409527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11359.725916] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11369.490107] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11377.519961] Lustre: Permanently deactivating lustre-OST0000 [11377.536141] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11387.980779] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11389.879488] LustreError: 393123:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11389.889246] LustreError: 393123:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11389.897396] LustreError: 393123:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11393.059213] LustreError: 393172:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11393.068163] LustreError: 393172:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11401.450201] Lustre: server umount lustre-OST0000 complete [11404.450240] LustreError: 393421:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11404.454860] LustreError: 393421:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11406.820023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11406.855412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11406.866877] Lustre: Skipped 1 previous similar message [11411.936653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11411.951156] Lustre: Skipped 1 previous similar message [11417.056749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11417.071659] Lustre: Skipped 1 previous similar message [11417.163654] Lustre: server umount lustre-MDT0000 complete [11421.772804] LustreError: 390109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872213 with bad export cookie 7142026299153506144 [11421.778058] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11421.797228] LustreError: 390109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11431.608662] Lustre: server umount lustre-MDT0001 complete [11442.421456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11448.181719] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11451.490141] LustreError: 394581:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11458.979357] Lustre: 394920:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.116@tcp [11461.568219] Lustre: server umount MGS complete [11474.514802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11474.998474] LustreError: 395644:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11475.025256] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11475.034071] Lustre: Skipped 1 previous similar message [11475.135346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11479.367810] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11485.666856] LustreError: 395645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.683618] LustreError: 395645:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11488.758074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11489.138153] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11495.707413] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11505.887458] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11512.514102] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11514.913987] Lustre: Permanently reactivating lustre-OST0000 [11514.943336] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11514.951974] Lustre: Skipped 2 previous similar messages [11544.036356] LustreError: 397438:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11552.054564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11552.489222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11552.500868] Lustre: Skipped 1 previous similar message [11558.894937] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11567.319827] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11574.250226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11574.252199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11574.257891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11574.281748] Lustre: Skipped 2 previous similar messages [11588.063230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11588.216458] Lustre: server umount lustre-OST0000 complete [11594.730660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11594.740412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11594.757396] Lustre: Skipped 1 previous similar message [11594.768917] Lustre: Skipped 6 previous similar messages [11598.386030] Lustre: server umount lustre-MDT0000 complete [11599.843670] LustreError: 395645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11602.497260] LustreError: 396381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872394 with bad export cookie 7142026299153507194 [11602.501867] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11602.514392] LustreError: 396381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11602.841768] Lustre: server umount lustre-MDT0001 complete [11614.497503] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11617.444989] Key type lgssc unregistered [11617.775977] LNet: 400021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11617.784298] LNetError: 400021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11617.813655] LNet: Removed LNI 192.168.203.116@tcp [11618.736417] Key type .llcrypt unregistered [11618.738888] Key type ._llcrypt unregistered [11638.415377] Key type ._llcrypt registered [11638.423192] Key type .llcrypt registered [11638.554931] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [11654.983248] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11656.202855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11656.246366] alg: No test for adler32 (adler32-zlib) [11657.315457] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [11657.586813] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11659.287591] Key type lgssc registered [11660.677856] Lustre: Echo OBD driver; http://www.lustre.org/ [11698.877155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11698.938213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11700.234254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11700.280480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11700.397383] Lustre: lustre-MDT0000: new disk, initializing [11700.529912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11700.553131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11705.422453] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11715.727605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11715.838994] Lustre: 403180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11715.934437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11715.940582] Lustre: Skipped 1 previous similar message [11716.077495] Lustre: lustre-MDT0001: new disk, initializing [11716.176527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11716.210792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11716.226621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11721.320955] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11731.328255] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11738.212555] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11746.321787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11746.645967] Lustre: lustre-OST0000: new disk, initializing [11746.651960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11746.667570] Lustre: 405149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11746.768369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11753.559282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11753.569821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11753.761241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11754.153511] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11765.028265] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11770.393629] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11770.599317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11776.637893] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11776.904449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11779.558855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11779.559190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11779.571464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11779.618705] Lustre: Skipped 1 previous similar message [11779.643326] Lustre: Skipped 1 previous similar message [11784.680030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11784.703200] Lustre: Skipped 1 previous similar message [11785.500656] Lustre: server umount lustre-OST0000 complete [11792.864686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11792.883316] LustreError: Skipped 1 previous similar message [11792.893625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11792.922210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11794.929205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.954643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11794.960619] Lustre: Skipped 1 previous similar message [11796.672389] Lustre: server umount lustre-MDT0000 complete [11800.298762] LustreError: 403173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872592 with bad export cookie 6345274399193771344 [11800.305818] LustreError: 403173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11800.312098] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11800.566909] Lustre: server umount lustre-MDT0001 complete [11810.040802] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11812.827028] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:16:42 (1787872602) [11821.586815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11822.178987] LustreError: 407608:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.195759] LustreError: 407608:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11822.254831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11827.688613] LustreError: 407609:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11827.884838] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11832.802358] LustreError: 407608:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11837.920511] LustreError: 407609:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11838.145382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11838.718603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11844.075270] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11852.637958] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11857.853713] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11865.059820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11865.450249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11872.084516] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11881.362764] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11883.777494] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11886.723072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11894.241234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11894.250023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11894.251126] Lustre: Skipped 1 previous similar message [11894.252686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11894.275353] Lustre: Skipped 1 previous similar message [11899.607660] Lustre: server umount lustre-OST0000 complete [11904.482170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11904.487859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11904.495131] Lustre: Skipped 1 previous similar message [11904.498331] Lustre: Skipped 2 previous similar messages [11909.611969] LustreError: 407608:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.642490] LustreError: 407608:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11909.743691] Lustre: server umount lustre-MDT0000 complete [11913.725109] LustreError: 407588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872705 with bad export cookie 6345274399193772226 [11913.729319] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11913.732089] LustreError: 407588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11914.645448] Lustre: server umount lustre-MDT0001 complete [11929.436598] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11932.627836] Key type lgssc unregistered [11932.933115] LNet: 411342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11932.937334] LNetError: 411342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11932.955728] LNet: Removed LNI 192.168.203.116@tcp [11933.652385] Key type .llcrypt unregistered [11933.654233] Key type ._llcrypt unregistered [11960.835313] Key type ._llcrypt registered [11960.836838] Key type .llcrypt registered [11960.955189] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11961.305140] Key type .llcrypt unregistered [11961.307746] Key type ._llcrypt unregistered [11978.151751] Key type ._llcrypt registered [11978.155277] Key type .llcrypt registered [11978.324922] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11979.613383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11979.798091] alg: No test for adler32 (adler32-zlib) [11980.960924] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [11981.364330] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11983.159871] Key type lgssc registered [11984.737467] Lustre: Echo OBD driver; http://www.lustre.org/ [11993.179788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11993.198478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11994.726976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11999.343675] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12004.324677] LustreError: 413562:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.350697] LustreError: 413562:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12007.733275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12008.120484] LustreError: 413561:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12008.143249] LustreError: 413561:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12008.148549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12013.542245] LustreError: 413557:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12013.561652] LustreError: 413557:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12014.146470] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12018.657597] LustreError: 413556:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.680268] LustreError: 413556:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12019.690899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12021.111324] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12024.803941] LustreError: 413561:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12024.820981] LustreError: 413561:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12026.882811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12027.283062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12032.724395] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12042.725300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12042.733861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12042.751870] Lustre: Skipped 1 previous similar message [12042.768935] Lustre: Skipped 1 previous similar message [12044.309644] Lustre: server umount lustre-OST0000 complete [12048.598575] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12048.615900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12048.645033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12052.961145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12052.964177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12055.027050] Lustre: server umount lustre-MDT0000 complete [12058.085517] LustreError: 414593:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12058.123620] LustreError: 414593:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12059.293797] LustreError: 415957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872851 with bad export cookie 12561408936536677906 [12059.297153] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12059.312816] LustreError: 415957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12059.562191] Lustre: server umount lustre-MDT0001 complete [12068.805327] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12071.276551] Key type lgssc unregistered [12071.596943] LNet: 416711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12071.614328] LNetError: 416711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12071.642593] LNet: Removed LNI 192.168.203.116@tcp [12072.571484] Key type .llcrypt unregistered [12072.573255] Key type ._llcrypt unregistered [12098.493972] Key type ._llcrypt registered [12098.498657] Key type .llcrypt registered [12098.655810] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12099.928405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12099.966162] alg: No test for adler32 (adler32-zlib) [12101.159742] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [12101.355460] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12103.071208] Key type lgssc registered [12104.456843] Lustre: Echo OBD driver; http://www.lustre.org/ [12113.367517] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12113.393695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12128.991565] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12134.111346] LustreError: 418101:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [12160.799256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12167.085695] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12173.330847] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12180.140327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12180.740865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12180.784611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12184.631204] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12186.082356] LustreError: 418125:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12186.097247] LustreError: 418125:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12191.209187] LustreError: 418126:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.599151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12193.130542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12197.904358] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12204.434331] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12209.173583] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12216.801102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12216.814793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12216.830068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12218.296184] Lustre: server umount lustre-OST0000 complete [12223.968899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12223.977540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12223.977986] LustreError: Skipped 1 previous similar message [12223.993361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12226.016480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12226.024153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12226.053589] Lustre: Skipped 1 previous similar message [12228.232343] Lustre: server umount lustre-MDT0000 complete [12231.136892] LustreError: 419138:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12231.151864] LustreError: 419138:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12231.674567] LustreError: 418109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873023 with bad export cookie 14074943221183321057 [12231.682956] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12231.693185] LustreError: 418109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12242.025238] Lustre: server umount lustre-MDT0001 complete [12253.410669] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12257.321904] Key type lgssc unregistered [12257.707954] LNet: 421676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12257.724744] LNetError: 421676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12257.753333] LNet: Removed LNI 192.168.203.116@tcp [12258.964183] Key type .llcrypt unregistered [12258.967806] Key type ._llcrypt unregistered [12285.392258] Key type ._llcrypt registered [12285.395512] Key type .llcrypt registered [12285.546856] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12286.639960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12286.727576] alg: No test for adler32 (adler32-zlib) [12287.861660] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [12288.134891] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12289.839206] Key type lgssc registered [12290.885993] Lustre: Echo OBD driver; http://www.lustre.org/ [12298.871407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12298.892974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.327124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12304.398770] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12310.499350] LustreError: 423094:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12310.515605] LustreError: 423094:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12312.356344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.760701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12316.691774] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12324.346389] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12329.855342] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12336.008603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12336.300440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12341.771199] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12345.533275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12349.209828] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12354.016564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12354.025341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12354.041992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12356.066331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.067267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12356.087245] Lustre: Skipped 1 previous similar message [12356.092262] Lustre: Skipped 1 previous similar message [12357.269968] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [12357.277645] Lustre: Skipped 1 previous similar message [12360.278285] Lustre: server umount lustre-MDT0000 complete [12361.189074] LustreError: 423824:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12361.220105] LustreError: 423824:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12362.707752] LustreError: 425672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.16@tcp arrived at 1787873154 with bad export cookie 11060707527143233982 [12362.730724] LustreError: 425672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12364.799387] LustreError: 423076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873156 with bad export cookie 11060707527143233086 [12364.807956] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12364.815985] LustreError: 423076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12366.306421] LustreError: 424149:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12366.306485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12366.344187] LustreError: 424149:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12366.379348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12369.312260] LustreError: 423094:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12369.327538] LustreError: 423094:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12371.077854] Lustre: server umount lustre-OST0000 complete [12374.436040] LustreError: 423093:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12374.464030] LustreError: 423093:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12377.408137] Lustre: server umount lustre-MDT0001 complete [12387.352329] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12390.926961] Key type lgssc unregistered [12391.291237] LNet: 426781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12391.301524] LNetError: 426781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12391.328862] LNet: Removed LNI 192.168.203.116@tcp [12392.402990] Key type .llcrypt unregistered [12392.407540] Key type ._llcrypt unregistered [12422.725149] Key type ._llcrypt registered [12422.728671] Key type .llcrypt registered [12422.819452] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12424.024057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12424.067170] alg: No test for adler32 (adler32-zlib) [12425.196189] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [12425.530513] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12427.240109] Key type lgssc registered [12428.519559] Lustre: Echo OBD driver; http://www.lustre.org/ [12437.093712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12437.110258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12438.622160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12443.277508] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12448.736040] LustreError: 428201:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12448.747256] LustreError: 428201:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12452.092582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12452.536771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12457.504347] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12466.039824] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12471.852565] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12479.425308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12479.855322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12483.951110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12486.149926] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12493.672794] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12497.462266] Lustre: server umount lustre-MDT0001 complete [12498.916155] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12498.928257] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12498.944236] LustreError: 428202:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12502.497290] LustreError: 428201:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12502.516151] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.545457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12504.552994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.573366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12507.618248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12507.620441] LustreError: 428929:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12507.651281] LustreError: 428929:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12509.898441] Lustre: server umount lustre-OST0000 complete [12512.739420] LustreError: 428202:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12514.249803] Lustre: server umount lustre-MDT0000 complete [12528.368239] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12531.094870] Key type lgssc unregistered [12531.367563] LNet: 431888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12531.376256] LNetError: 431888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12531.390256] LNet: Removed LNI 192.168.203.116@tcp [12532.257807] Key type .llcrypt unregistered [12532.262722] Key type ._llcrypt unregistered [12561.109938] Key type ._llcrypt registered [12561.128437] Key type .llcrypt registered [12561.309806] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [12577.690064] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12578.859719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12578.903437] alg: No test for adler32 (adler32-zlib) [12580.671393] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [12581.107675] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12582.783410] Key type lgssc registered [12584.729408] Lustre: Echo OBD driver; http://www.lustre.org/ [12629.172693] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12645.333891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12645.355662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12646.596883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12646.652555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12646.752130] Lustre: lustre-MDT0000: new disk, initializing [12646.930653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12646.961487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12651.886732] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12668.612685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12668.952677] Lustre: lustre-OST0000: new disk, initializing [12668.957721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12668.966356] Lustre: Skipped 1 previous similar message [12668.972920] Lustre: 436927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12669.059620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12675.939962] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12677.263500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12677.278291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12677.323563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12685.827429] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12702.288616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12702.502703] Lustre: 435907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12702.674769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12702.822371] Lustre: lustre-MDT0001: new disk, initializing [12703.004169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12703.053922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12703.065804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12708.464793] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12717.640869] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12717.964394] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12722.367472] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12722.646348] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12727.784760] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12727.788355] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12727.801690] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12727.831291] Lustre: Skipped 1 previous similar message [12728.966416] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [12732.899996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12732.910174] Lustre: Skipped 1 previous similar message [12733.969092] Lustre: server umount lustre-MDT0001 complete [12743.139645] LustreError: 435914:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.159112] LustreError: 435914:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12748.265972] LustreError: 435919:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12748.298565] LustreError: 435919:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12753.377206] LustreError: 435920:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12753.380299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.430219] Lustre: Skipped 1 previous similar message [12753.438909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12757.956984] Lustre: server umount lustre-OST0000 complete [12758.510394] LustreError: 435919:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12768.253076] Lustre: server umount lustre-MDT0000 complete [12783.409838] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12786.258800] Key type lgssc unregistered [12786.634807] LNet: 440680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12786.642524] LNetError: 440680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12786.673768] LNet: Removed LNI 192.168.203.116@tcp [12787.646213] Key type .llcrypt unregistered [12787.647895] Key type ._llcrypt unregistered [12829.283595] Key type ._llcrypt registered [12829.286323] Key type .llcrypt registered [12829.378382] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12829.685715] Key type .llcrypt unregistered [12829.687428] Key type ._llcrypt unregistered [12848.482427] Key type ._llcrypt registered [12848.484829] Key type .llcrypt registered [12848.600064] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12849.651688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12849.671967] alg: No test for adler32 (adler32-zlib) [12850.749608] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [12851.022029] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12852.719170] Key type lgssc registered [12853.728456] Lustre: Echo OBD driver; http://www.lustre.org/ [12881.192433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12881.227394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12882.540331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12882.582323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12882.710288] Lustre: lustre-MDT0000: new disk, initializing [12882.811456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12882.842517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12887.481180] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12898.219670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12898.344973] Lustre: 444062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12898.448189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12898.455543] Lustre: Skipped 1 previous similar message [12898.532832] Lustre: lustre-MDT0001: new disk, initializing [12898.683859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12898.734807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12898.755149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12903.603435] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12913.822566] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12920.220542] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12927.835896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12928.109556] Lustre: lustre-OST0000: new disk, initializing [12928.113473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12928.119382] Lustre: 446035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12928.211829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12934.237109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12934.250651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12934.370306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12934.843136] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12944.998773] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12955.111775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12955.114264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12955.128420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12955.136473] Lustre: Skipped 2 previous similar messages [12955.154370] Lustre: Skipped 2 previous similar messages [12957.555758] Lustre: server umount lustre-MDT0000 complete [12961.503292] LustreError: 444053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873753 with bad export cookie 15362096139720706490 [12961.505498] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12961.513762] LustreError: 444053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12961.840561] Lustre: server umount lustre-MDT0001 complete [12976.285522] Lustre: server umount lustre-OST0000 complete [12987.538876] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:36:17 (1787873777) [12995.564970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12996.021971] LustreError: 448064:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12996.043017] LustreError: 448064:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12996.100359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13000.727604] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13001.185358] LustreError: 448065:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13005.302370] LustreError: 448064:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13009.164924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13009.555351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13013.406692] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13020.537843] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13028.214905] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13048.173141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13048.446621] Lustre: Found index 0 for lustre-OST0000, updating log [13048.702613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13055.960347] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13065.712640] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13068.146439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13078.501166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13078.508567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13078.511421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13078.528707] Lustre: Skipped 1 previous similar message [13078.569796] Lustre: Skipped 1 previous similar message [13079.652768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13079.668159] Lustre: Skipped 1 previous similar message [13082.574748] Lustre: server umount lustre-OST0000 complete [13101.537081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13101.539902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13101.544928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13101.574945] Lustre: Skipped 1 previous similar message [13104.548668] Lustre: server umount lustre-MDT0000 complete [13106.656165] LustreError: 448064:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13106.679796] LustreError: 448064:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13108.251576] LustreError: 448044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873900 with bad export cookie 15362096139720707876 [13108.277581] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13108.781681] Lustre: server umount lustre-MDT0001 complete [13115.232226] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [13117.469535] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:38:27 (1787873907) [13735.542782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13735.976781] LustreError: 454473:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13735.993398] LustreError: 454473:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13736.061043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13740.820914] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13741.537376] LustreError: 454474:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13741.555064] LustreError: 454474:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13745.631737] LustreError: 454473:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13749.242037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13749.620691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13754.264846] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13762.318468] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13768.764923] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13788.712785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13788.971524] Lustre: Found index 0 for lustre-OST0000, updating log [13789.346730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13793.119760] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [13796.915579] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13806.780762] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13809.157933] Lustre: server umount lustre-OST0000 complete [13813.865970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13813.876485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13813.877376] Lustre: Skipped 1 previous similar message [13813.890645] Lustre: Skipped 2 previous similar messages [13814.945352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13814.957392] Lustre: Skipped 1 previous similar message [13815.968362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13815.974244] Lustre: Skipped 1 previous similar message [13818.081497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13818.090534] Lustre: Skipped 3 previous similar messages [13819.105722] LustreError: 455202:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13819.115864] LustreError: 455202:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13819.232925] Lustre: server umount lustre-MDT0000 complete [13823.204359] LustreError: 454473:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13823.233728] LustreError: 454473:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [13823.468333] LustreError: 454454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874615 with bad export cookie 15362096139720708968 [13823.483421] LustreError: 454454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13823.485212] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13823.995364] Lustre: server umount lustre-MDT0001 complete [13852.513604] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [13857.700120] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [13867.652621] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13908.656372] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:51:37 (1787874697) [13928.263461] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [13931.761745] Key type lgssc unregistered [13932.112629] LNet: 461088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13932.128612] LNetError: 461088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14050.922567] LNet: Removed LNI 192.168.203.116@tcp [14051.852864] Key type .llcrypt unregistered [14051.856388] Key type ._llcrypt unregistered [14069.145786] Key type ._llcrypt registered [14069.150214] Key type .llcrypt registered [14069.262270] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14070.439505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14070.688841] alg: No test for adler32 (adler32-zlib) [14071.811923] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [14072.115971] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [14073.791205] Key type lgssc registered [14074.939366] Lustre: Echo OBD driver; http://www.lustre.org/ [14079.086384] LNet: Added LNI 192.168.203.116@tcp100 [8/256/0/180] [14079.977706] LNet: Added LNI 192.168.203.116@tcp101 [8/256/0/180] [14080.748455] LNet: Added LNI 192.168.203.116@tcp102 [8/256/0/180] [14081.673382] LNet: Added LNI 192.168.203.116@tcp103 [8/256/0/180] [14082.530660] LNet: Added LNI 192.168.203.116@tcp104 [8/256/0/180] [14083.410233] LNet: Added LNI 192.168.203.116@tcp105 [8/256/0/180] [14084.248831] LNet: Added LNI 192.168.203.116@tcp106 [8/256/0/180] [14085.130403] LNet: Added LNI 192.168.203.116@tcp107 [8/256/0/180] [14086.045771] LNet: Added LNI 192.168.203.116@tcp108 [8/256/0/180] [14086.907230] LNet: Added LNI 192.168.203.116@tcp109 [8/256/0/180] [14087.677796] LNet: Added LNI 192.168.203.116@tcp110 [8/256/0/180] [14088.618353] LNet: Added LNI 192.168.203.116@tcp111 [8/256/0/180] [14089.494971] LNet: Added LNI 192.168.203.116@tcp112 [8/256/0/180] [14090.584810] LNet: Added LNI 192.168.203.116@tcp113 [8/256/0/180] [14091.464620] LNet: Added LNI 192.168.203.116@tcp114 [8/256/0/180] [14092.354689] LNet: Added LNI 192.168.203.116@tcp115 [8/256/0/180] [14093.147677] LNet: Added LNI 192.168.203.116@tcp116 [8/256/0/180] [14093.926616] LNet: Added LNI 192.168.203.116@tcp117 [8/256/0/180] [14094.760777] LNet: Added LNI 192.168.203.116@tcp118 [8/256/0/180] [14095.609703] LNet: Added LNI 192.168.203.116@tcp119 [8/256/0/180] [14096.574566] LNet: Added LNI 192.168.203.116@tcp120 [8/256/0/180] [14097.512902] LNet: Added LNI 192.168.203.116@tcp121 [8/256/0/180] [14098.436022] LNet: Added LNI 192.168.203.116@tcp122 [8/256/0/180] [14099.373483] LNet: Added LNI 192.168.203.116@tcp123 [8/256/0/180] [14100.278820] LNet: Added LNI 192.168.203.116@tcp124 [8/256/0/180] [14101.165665] LNet: Added LNI 192.168.203.116@tcp125 [8/256/0/180] [14101.932813] LNet: Added LNI 192.168.203.116@tcp126 [8/256/0/180] [14102.793565] LNet: Added LNI 192.168.203.116@tcp127 [8/256/0/180] [14103.700215] LNet: Added LNI 192.168.203.116@tcp128 [8/256/0/180] [14104.482967] LNet: Added LNI 192.168.203.116@tcp129 [8/256/0/180] [14105.217445] LNet: Added LNI 192.168.203.116@tcp130 [8/256/0/180] [14106.106518] LNet: Added LNI 192.168.203.116@tcp131 [8/256/0/180] [14106.824387] LNet: Added LNI 192.168.203.116@tcp132 [8/256/0/180] [14107.774896] LNet: Added LNI 192.168.203.116@tcp133 [8/256/0/180] [14108.662101] LNet: Added LNI 192.168.203.116@tcp134 [8/256/0/180] [14109.430523] LNet: Added LNI 192.168.203.116@tcp135 [8/256/0/180] [14116.118811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14116.144324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14117.304877] Lustre: 464119:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [14117.384147] LNetError: 464144:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [14117.397330] LNetError: Protocol error connecting to 192.168.203.116@tcp122 at host 192.168.203.116:988: is it running a compatible version of Lustre? [14117.411808] LNetError: Skipped 6 previous similar messages [14117.526314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14117.576088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14117.662690] Lustre: lustre-MDT0000: new disk, initializing [14117.723469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14117.739024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14118.239555] LNet: 6 local NIs in recovery (showing 6): 192.168.203.116@tcp125, 192.168.203.116@tcp126, 192.168.203.116@tcp128, 192.168.203.116@tcp129, 192.168.203.116@tcp127, 192.168.203.116@tcp122 [14118.240602] LNetError: 461581:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [14118.289779] LNetError: 461581:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14118.303194] LNetError: Protocol error connecting to 192.168.203.116@tcp104 at host 192.168.203.116:988: is it running a compatible version of Lustre? [14118.319315] LNetError: Skipped 2 previous similar messages [14122.317561] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14132.103326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14132.261526] Lustre: 464138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14132.314405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14132.322778] Lustre: Skipped 1 previous similar message [14132.425831] Lustre: lustre-MDT0001: new disk, initializing [14132.463153] Lustre: 464943:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [14132.482555] Lustre: 464943:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14132.604448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14132.661367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14132.669515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14135.237395] Lustre: 465078:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [14135.251779] Lustre: 465078:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14136.942503] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14147.100574] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14147.875493] LNetError: 464140:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [14147.886645] LNetError: Protocol error connecting to 192.168.203.116@tcp119 at host 192.168.203.116:988: is it running a compatible version of Lustre? [14150.948235] LNetError: 464150:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [14150.967340] LNetError: Protocol error connecting to 192.168.203.116@tcp104 at host 192.168.203.116:988: is it running a compatible version of Lustre? [14154.804959] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14158.191460] LNetError: 464144:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [14158.211139] LNetError: 464144:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14158.230167] LNetError: Protocol error connecting to 192.168.203.116@tcp124 at host 192.168.203.116:988: is it running a compatible version of Lustre? [14158.259604] LNetError: Skipped 2 previous similar messages [14162.545604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14163.141634] Lustre: lustre-OST0000: new disk, initializing [14163.150603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14163.156911] Lustre: 466115:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14163.367462] Lustre: 466120:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [14163.382146] Lustre: 466120:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14163.475925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14166.372955] LNetError: 464147:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [14166.385480] LNetError: 464147:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14166.392216] LNetError: Protocol error connecting to 192.168.203.116@tcp123 at host 192.168.203.116:988: is it running a compatible version of Lustre? [14166.403921] LNetError: Skipped 3 previous similar messages [14170.382765] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14170.696593] Lustre: 466485:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [14170.704900] Lustre: 466485:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14170.764869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14170.770760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14170.869161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14179.862850] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14184.802600] LNetError: 464155:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [14184.823129] LNetError: 464155:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14184.843744] LNetError: Protocol error connecting to 192.168.203.116@tcp124 at host 192.168.203.116:988: is it running a compatible version of Lustre? [14184.857119] LNetError: Skipped 5 previous similar messages [14191.585740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14191.598691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14191.608566] Lustre: Skipped 1 previous similar message [14191.616791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14191.650628] Lustre: Skipped 1 previous similar message [14193.861435] Lustre: server umount lustre-OST0000 complete [14199.269183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14199.286685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14199.315277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14201.836359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14201.838101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14201.859686] Lustre: Skipped 1 previous similar message [14203.826186] Lustre: server umount lustre-MDT0000 complete [14207.255214] LustreError: 464942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874999 with bad export cookie 15049568540290479971 [14207.259751] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14207.263195] LustreError: 464942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14207.506268] Lustre: server umount lustre-MDT0001 complete [14217.048763] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [14217.573272] LNetError: 464141:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.116 [14217.582432] LNetError: 464141:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14217.587096] LNetError: Protocol error connecting to 192.168.203.116@tcp123 at host 192.168.203.116:988: is it running a compatible version of Lustre? [14217.594668] LNetError: Skipped 7 previous similar messages [14219.577974] Key type lgssc unregistered [14219.932779] LNet: 468030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14219.944024] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14219.950967] LNet: Removed LNI 192.168.203.116@tcp [14220.959184] LNet: Removed LNI 192.168.203.116@tcp100 [14220.967214] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14220.977895] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14220.984481] LNet: Removed LNI 192.168.203.116@tcp101 [14222.047430] LNet: Removed LNI 192.168.203.116@tcp102 [14222.056475] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14222.065449] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14222.071474] LNet: Removed LNI 192.168.203.116@tcp103 [14223.136482] LNet: Removed LNI 192.168.203.116@tcp104 [14224.159174] LNet: Removed LNI 192.168.203.116@tcp105 [14224.165228] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14224.169482] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14224.174671] LNet: Removed LNI 192.168.203.116@tcp106 [14225.183106] LNet: Removed LNI 192.168.203.116@tcp107 [14226.207377] LNet: Removed LNI 192.168.203.116@tcp108 [14227.231173] LNet: Removed LNI 192.168.203.116@tcp109 [14228.255327] LNet: Removed LNI 192.168.203.116@tcp110 [14228.258441] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14228.263697] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14228.270851] LNet: Removed LNI 192.168.203.116@tcp111 [14229.279149] LNet: Removed LNI 192.168.203.116@tcp112 [14230.303159] LNet: Removed LNI 192.168.203.116@tcp113 [14231.329094] LNet: Removed LNI 192.168.203.116@tcp114 [14232.351187] LNet: Removed LNI 192.168.203.116@tcp115 [14233.375298] LNet: Removed LNI 192.168.203.116@tcp116 [14234.399141] LNet: Removed LNI 192.168.203.116@tcp117 [14235.423373] LNet: Removed LNI 192.168.203.116@tcp118 [14236.447359] LNet: Removed LNI 192.168.203.116@tcp119 [14236.456526] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14236.467584] LNetError: 468030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14236.477937] LNet: Removed LNI 192.168.203.116@tcp120 [14237.536082] LNet: Removed LNI 192.168.203.116@tcp121 [14238.559227] LNet: Removed LNI 192.168.203.116@tcp122 [14239.583177] LNet: Removed LNI 192.168.203.116@tcp123 [14240.607168] LNet: Removed LNI 192.168.203.116@tcp124 [14241.631162] LNet: Removed LNI 192.168.203.116@tcp125 [14242.655134] LNet: Removed LNI 192.168.203.116@tcp126 [14243.679506] LNet: Removed LNI 192.168.203.116@tcp127 [14244.703235] LNet: Removed LNI 192.168.203.116@tcp128 [14245.727201] LNet: Removed LNI 192.168.203.116@tcp129 [14245.736644] LNet: Removed LNI 192.168.203.116@tcp130 [14245.741847] LNet: Removed LNI 192.168.203.116@tcp131 [14245.745837] LNet: Removed LNI 192.168.203.116@tcp132 [14245.750957] LNet: Removed LNI 192.168.203.116@tcp133 [14245.754210] LNet: Removed LNI 192.168.203.116@tcp134 [14245.763610] LNet: Removed LNI 192.168.203.116@tcp135 [14246.435314] Key type .llcrypt unregistered [14246.437493] Key type ._llcrypt unregistered [14267.508758] Key type ._llcrypt registered [14267.511674] Key type .llcrypt registered [14267.615585] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [14282.402736] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14283.907877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14283.965261] alg: No test for adler32 (adler32-zlib) [14285.080403] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [14285.355789] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [14287.031696] Key type lgssc registered [14288.310817] Lustre: Echo OBD driver; http://www.lustre.org/ [14324.849200] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:58:34 (1787875114) [14342.397164] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [14345.866153] Key type lgssc unregistered [14346.263518] LNet: 472123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14346.279854] LNetError: 472123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14346.309707] LNet: Removed LNI 192.168.203.116@tcp [14347.130375] Key type .llcrypt unregistered [14347.132430] Key type ._llcrypt unregistered [14360.302722] Key type ._llcrypt registered [14360.304867] Key type .llcrypt registered [14360.376388] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14361.165903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14361.252658] alg: No test for adler32 (adler32-zlib) [14362.258495] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [14362.540743] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [14364.199173] Key type lgssc registered [14365.018662] Lustre: Echo OBD driver; http://www.lustre.org/ [14373.603971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14373.625300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14374.852602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14374.926398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14375.021560] Lustre: lustre-MDT0000: new disk, initializing [14375.094221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14375.122608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14378.771483] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14387.953398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14388.058911] Lustre: 473315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14388.135372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14388.138872] Lustre: Skipped 1 previous similar message [14388.218940] Lustre: lustre-MDT0001: new disk, initializing [14388.315837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14388.348097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14388.361689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14392.212964] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14400.816506] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14406.312395] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14412.681138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14412.846945] Lustre: lustre-OST0000: new disk, initializing [14412.849410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14412.854734] Lustre: 475282:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14412.934060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14418.302425] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14418.973903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14418.986244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14419.110674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14426.890548] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14430.121714] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing lnet_if_list [14434.177178] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14434.194848] LNet: Added LNI 192.168.203.116@tcp10 [8/256/0/180] [14439.304520] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14439.329547] LNet: Added LNI 192.168.203.116@tcp11 [8/256/0/180] [14442.802204] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14442.819428] LNet: Added LNI 192.168.203.116@tcp12 [8/256/0/180] [14446.391937] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14446.417434] LNet: Added LNI 192.168.203.116@tcp13 [8/256/0/180] [14451.364897] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14451.385220] LNet: Added LNI 192.168.203.116@tcp14 [8/256/0/180] [14455.158173] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14455.191483] LNet: Added LNI 192.168.203.116@tcp15 [8/256/0/180] [14460.373820] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14460.450583] LNet: Added LNI 192.168.203.116@tcp16 [8/256/0/180] [14466.112307] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14466.141146] LNet: Added LNI 192.168.203.116@tcp17 [8/256/0/180] [14472.534559] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14472.551423] LNet: Added LNI 192.168.203.116@tcp18 [8/256/0/180] [14479.519891] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14479.546470] LNet: Added LNI 192.168.203.116@tcp19 [8/256/0/180] [14489.321996] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14489.484313] LNet: Added LNI 192.168.203.116@tcp20 [8/256/0/180] [14501.819081] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14501.872831] LNet: Added LNI 192.168.203.116@tcp21 [8/256/0/180] [14512.495349] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14512.582107] LNet: Added LNI 192.168.203.116@tcp22 [8/256/0/180] [14520.785754] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14520.832567] LNet: Added LNI 192.168.203.116@tcp23 [8/256/0/180] [14530.952591] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14531.018798] LNet: Added LNI 192.168.203.116@tcp24 [8/256/0/180] [14540.838705] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14540.870785] LNet: Added LNI 192.168.203.116@tcp25 [8/256/0/180] [14549.432934] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14549.468762] LNet: Added LNI 192.168.203.116@tcp26 [8/256/0/180] [14555.544160] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14555.569314] LNet: Added LNI 192.168.203.116@tcp27 [8/256/0/180] [14561.677311] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14561.711228] LNet: Added LNI 192.168.203.116@tcp28 [8/256/0/180] [14567.890572] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14567.917659] LNet: Added LNI 192.168.203.116@tcp29 [8/256/0/180] [14573.958469] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14573.985879] LNet: Added LNI 192.168.203.116@tcp30 [8/256/0/180] [14580.071275] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14580.109444] LNet: Added LNI 192.168.203.116@tcp31 [8/256/0/180] [14586.554118] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14586.572396] LNet: Added LNI 192.168.203.116@tcp32 [8/256/0/180] [14592.694923] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14592.743436] LNet: Added LNI 192.168.203.116@tcp33 [8/256/0/180] [14598.993593] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14599.017996] LNet: Added LNI 192.168.203.116@tcp34 [8/256/0/180] [14605.778840] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14605.800420] LNet: Added LNI 192.168.203.116@tcp35 [8/256/0/180] [14612.343394] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14612.381352] LNet: Added LNI 192.168.203.116@tcp36 [8/256/0/180] [14620.147032] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14620.185343] LNet: Added LNI 192.168.203.116@tcp37 [8/256/0/180] [14628.432899] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14628.468491] LNet: Added LNI 192.168.203.116@tcp38 [8/256/0/180] [14635.662552] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14635.697759] LNet: Added LNI 192.168.203.116@tcp39 [8/256/0/180] [14643.835812] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14643.881543] LNet: Added LNI 192.168.203.116@tcp40 [8/256/0/180] [14652.281464] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14652.319795] LNet: Added LNI 192.168.203.116@tcp41 [8/256/0/180] [14660.416235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14660.460936] LNet: Added LNI 192.168.203.116@tcp42 [8/256/0/180] [14669.027684] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14669.069416] LNet: Added LNI 192.168.203.116@tcp43 [8/256/0/180] [14676.565664] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14676.613614] LNet: Added LNI 192.168.203.116@tcp44 [8/256/0/180] [14683.589360] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14683.611956] LNet: Added LNI 192.168.203.116@tcp45 [8/256/0/180] [14691.751870] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14691.786911] LNet: Added LNI 192.168.203.116@tcp46 [8/256/0/180] [14699.263755] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14699.313310] LNet: Added LNI 192.168.203.116@tcp47 [8/256/0/180] [14707.275192] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14707.315735] LNet: Added LNI 192.168.203.116@tcp48 [8/256/0/180] [14714.804849] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14714.846989] LNet: Added LNI 192.168.203.116@tcp49 [8/256/0/180] [14722.672838] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14722.716486] LNet: Added LNI 192.168.203.116@tcp50 [8/256/0/180] [14741.473842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14741.480551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14741.491088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14741.986362] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14741.996307] Lustre: Skipped 1 previous similar message [14741.999984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14742.006145] Lustre: Skipped 1 previous similar message [14747.105508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14747.118200] Lustre: Skipped 2 previous similar messages [14752.227110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14752.235740] Lustre: Skipped 2 previous similar messages [14754.272277] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14754.493296] Lustre: server umount lustre-MDT0000 complete [14758.270928] LustreError: 475281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875550 with bad export cookie 6687293971369754173 [14758.279665] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14758.286102] LustreError: 475281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14758.553805] Lustre: server umount lustre-MDT0001 complete [14776.879597] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14778.784333] Lustre: 472587:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787875554/real 1787875554] req@0000000055e20013 x1874722977654912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787875570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14778.813177] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14779.871617] Lustre: 472589:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787875555/real 1787875555] req@000000005025c75f x1874722977655040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787875571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14784.033218] Lustre: 472589:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787875559/real 1787875559] req@000000007198abfe x1874722977655168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787875575 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14785.453841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14790.111579] Lustre: 472586:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787875565/real 1787875565] req@00000000bdcf35dc x1874722977655552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787875581 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14790.141170] Lustre: 472586:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14803.423520] LustreError: 472585:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000035f2238a x1874722977656192/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14803.630894] LustreError: 475275:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14803.640203] LustreError: 475275:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14803.672425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14808.284591] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14809.060877] LustreError: 475276:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14813.168153] LustreError: 475275:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14813.170536] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14817.043466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14817.542811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14821.972210] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14823.933847] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14830.642663] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14836.778414] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14853.605375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14853.612747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14853.622250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14864.869153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14864.874583] Lustre: Skipped 5 previous similar messages [14865.377070] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14865.544739] Lustre: server umount lustre-OST0000 complete [14869.985830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14870.005538] Lustre: Skipped 1 previous similar message [14875.729082] Lustre: server umount lustre-MDT0000 complete [14880.257652] LustreError: 499539:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14880.284950] LustreError: 499539:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14881.636996] LustreError: 473307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875673 with bad export cookie 6687293971369756798 [14881.638772] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14881.661456] LustreError: 473307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14882.123435] Lustre: server umount lustre-MDT0001 complete [14896.400315] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [14900.058765] Key type lgssc unregistered [14900.428367] LNet: 502324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14900.436212] LNetError: 502324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14900.444512] LNet: Removed LNI 192.168.203.116@tcp [14900.447898] LNet: Removed LNI 192.168.203.116@tcp10 [14900.451585] LNet: Removed LNI 192.168.203.116@tcp11 [14900.455254] LNet: Removed LNI 192.168.203.116@tcp12 [14900.458171] LNet: Removed LNI 192.168.203.116@tcp13 [14900.468277] LNet: Removed LNI 192.168.203.116@tcp14 [14900.477295] LNet: Removed LNI 192.168.203.116@tcp15 [14900.480849] LNet: Removed LNI 192.168.203.116@tcp16 [14900.487095] LNet: Removed LNI 192.168.203.116@tcp17 [14900.491175] LNet: Removed LNI 192.168.203.116@tcp18 [14900.497247] LNet: Removed LNI 192.168.203.116@tcp19 [14900.499392] LNet: Removed LNI 192.168.203.116@tcp20 [14900.505668] LNet: Removed LNI 192.168.203.116@tcp21 [14900.508252] LNet: Removed LNI 192.168.203.116@tcp22 [14900.512965] LNet: Removed LNI 192.168.203.116@tcp23 [14900.515851] LNet: Removed LNI 192.168.203.116@tcp24 [14900.518628] LNet: Removed LNI 192.168.203.116@tcp25 [14900.522492] LNet: Removed LNI 192.168.203.116@tcp26 [14900.533324] LNet: Removed LNI 192.168.203.116@tcp27 [14900.539816] LNet: Removed LNI 192.168.203.116@tcp28 [14900.543669] LNet: Removed LNI 192.168.203.116@tcp29 [14900.552366] LNet: Removed LNI 192.168.203.116@tcp30 [14900.556500] LNet: Removed LNI 192.168.203.116@tcp31 [14900.571168] LNet: Removed LNI 192.168.203.116@tcp32 [14900.582454] LNet: Removed LNI 192.168.203.116@tcp33 [14900.587355] LNet: Removed LNI 192.168.203.116@tcp34 [14900.593131] LNet: Removed LNI 192.168.203.116@tcp35 [14900.598443] LNet: Removed LNI 192.168.203.116@tcp36 [14900.601650] LNet: Removed LNI 192.168.203.116@tcp37 [14900.604510] LNet: Removed LNI 192.168.203.116@tcp38 [14900.608874] LNet: Removed LNI 192.168.203.116@tcp39 [14900.612606] LNet: Removed LNI 192.168.203.116@tcp40 [14900.617782] LNet: Removed LNI 192.168.203.116@tcp41 [14900.620865] LNet: Removed LNI 192.168.203.116@tcp42 [14900.623849] LNet: Removed LNI 192.168.203.116@tcp43 [14900.627955] LNet: Removed LNI 192.168.203.116@tcp44 [14900.631303] LNet: Removed LNI 192.168.203.116@tcp45 [14900.634917] LNet: Removed LNI 192.168.203.116@tcp46 [14900.639521] LNet: Removed LNI 192.168.203.116@tcp47 [14900.642797] LNet: Removed LNI 192.168.203.116@tcp48 [14900.647067] LNet: Removed LNI 192.168.203.116@tcp49 [14900.661531] LNet: Removed LNI 192.168.203.116@tcp50 [14901.511482] Key type .llcrypt unregistered [14901.518214] Key type ._llcrypt unregistered [14922.008853] Key type ._llcrypt registered [14922.011841] Key type .llcrypt registered [14922.103689] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [14936.402523] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14937.222382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14937.327269] alg: No test for adler32 (adler32-zlib) [14938.548216] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [14938.920956] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [14940.671208] Key type lgssc registered [14942.384238] Lustre: Echo OBD driver; http://www.lustre.org/ [14977.971346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14977.983622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14979.191242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14979.212423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14979.286096] Lustre: lustre-MDT0000: new disk, initializing [14979.415292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14979.448606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14984.286898] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14996.677396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14996.885251] Lustre: 505520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14997.035897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14997.048454] Lustre: Skipped 1 previous similar message [14997.250383] Lustre: lustre-MDT0001: new disk, initializing [14997.463258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14997.547422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14997.561166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15004.176193] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15014.991495] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15021.196578] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15028.517459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15028.749564] Lustre: *** cfs_fail_loc=90f, val=0*** [15028.827734] Lustre: lustre-OST0000: new disk, initializing [15028.834505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15028.841631] Lustre: 507540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15028.935387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15035.023826] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15037.509070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15037.520516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15037.625070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15044.458866] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15053.283390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15053.292511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15053.294305] LustreError: Skipped 1 previous similar message [15053.296865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15053.312215] Lustre: Skipped 1 previous similar message [15053.324064] Lustre: Skipped 1 previous similar message [15057.520163] Lustre: server umount lustre-OST0000 complete [15063.522591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15063.535401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15068.645884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15068.656083] Lustre: Skipped 2 previous similar messages [15073.763931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15073.770255] Lustre: Skipped 1 previous similar message [15075.295813] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15075.502051] Lustre: server umount lustre-MDT0000 complete [15078.825477] LustreError: 505514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875870 with bad export cookie 9138611681183912601 [15078.826365] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15078.845164] LustreError: 505514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15079.163813] Lustre: server umount lustre-MDT0001 complete [15088.806286] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [15091.563490] Key type lgssc unregistered [15091.863926] LNet: 509450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15091.886225] LNetError: 509450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15091.901206] LNet: Removed LNI 192.168.203.116@tcp [15092.806345] Key type .llcrypt unregistered [15092.809792] Key type ._llcrypt unregistered [15115.115965] Key type ._llcrypt registered [15115.120791] Key type .llcrypt registered [15115.332767] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [15116.620579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15116.683415] alg: No test for adler32 (adler32-zlib) [15117.704865] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [15117.996387] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [15119.719437] Key type lgssc registered [15120.911552] Lustre: Echo OBD driver; http://www.lustre.org/ [15130.079578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15130.116635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15131.895528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15137.845692] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15141.345985] LustreError: 510722:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15141.359854] LustreError: 510722:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15146.468820] LustreError: 510723:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15148.337577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15148.787778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15154.752133] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15165.035199] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15173.530719] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15182.953104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15183.250298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15190.347656] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15202.257935] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15212.896333] loop0: detected capacity change from 0 to 204800000 [15213.239702] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15213.256544] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15213.267855] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15222.142625] loop0: detected capacity change from 0 to 204800000 [15222.439069] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15222.468182] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15222.504571] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15228.514797] loop0: detected capacity change from 0 to 204800000 [15235.050662] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15235.165879] Lustre: fs15246-OST0000: new disk, initializing [15235.169557] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15235.175240] Lustre: 514317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15235.240613] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15242.058402] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15251.803155] loop1: detected capacity change from 0 to 204800000 [15257.003533] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15257.058130] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15257.098491] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15257.166775] Lustre: fs15246-MDT0000: new disk, initializing [15257.262511] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15257.296614] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15257.386992] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15258.405582] Lustre: 515441:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000045b98559 x1874723769208832/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787876066 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15260.343595] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15267.505950] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15267.515834] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15267.545890] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15280.633525] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15280.662542] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15285.745152] LustreError: 511820:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15285.912832] Lustre: server umount fs15246-MDT0000 complete [15290.860405] LustreError: 510718:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15295.972935] LustreError: 510718:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15296.432823] Lustre: server umount fs15246-OST0000 complete [15311.334393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15311.353652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15311.354218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15311.370964] Lustre: Skipped 1 previous similar message [15316.453566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15316.459953] Lustre: Skipped 1 previous similar message [15321.571753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15321.582189] Lustre: Skipped 1 previous similar message [15324.127272] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15324.260184] Lustre: server umount lustre-OST0000 complete [15328.224804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15328.229244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15328.239688] Lustre: Skipped 1 previous similar message [15328.246757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15331.814714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15333.897042] Lustre: server umount lustre-MDT0000 complete [15336.928648] LustreError: 510719:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15336.959331] LustreError: 510719:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15337.403455] LustreError: 510704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876129 with bad export cookie 13641555439085883354 [15337.406591] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15337.417434] LustreError: 510704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15337.662876] Lustre: server umount lustre-MDT0001 complete [15351.207292] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [15355.523739] Key type lgssc unregistered [15356.029719] LNet: 518543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15356.051423] LNetError: 518543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15356.103809] LNet: Removed LNI 192.168.203.116@tcp [15357.277897] Key type .llcrypt unregistered [15357.280670] Key type ._llcrypt unregistered [15380.954651] Key type ._llcrypt registered [15380.957180] Key type .llcrypt registered [15381.052316] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [15382.352489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15382.387165] alg: No test for adler32 (adler32-zlib) [15383.589745] Lustre: Lustre: Build Version: 2.17.57_120_gb73d7ca [15383.846369] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [15385.535179] Key type lgssc registered [15386.693182] Lustre: Echo OBD driver; http://www.lustre.org/ [15426.256331] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [15439.415830] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [15481.358044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15481.395516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15482.597134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15482.658564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15482.808342] Lustre: lustre-MDT0000: new disk, initializing [15482.923110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15482.938180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15488.017804] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15498.200888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15498.330685] Lustre: 523028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15498.401936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15498.411430] Lustre: Skipped 1 previous similar message [15498.428440] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15498.494911] Lustre: lustre-MDT0001: new disk, initializing [15498.588931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15498.636566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15498.653955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15502.289696] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15512.158834] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15517.917429] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15524.291506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15524.708187] Lustre: lustre-OST0000: new disk, initializing [15524.712563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15524.718172] Lustre: 524998:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15524.815056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15531.609493] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15532.672328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15532.683717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15532.875374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15542.714410] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15549.015818] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15549.266655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15553.417684] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15553.617177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15558.629965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15558.642820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15558.668750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15561.776690] Lustre: server umount lustre-OST0000 complete [15568.869826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15568.873939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15568.884928] Lustre: Skipped 2 previous similar messages [15568.898954] Lustre: Skipped 2 previous similar messages [15573.984988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15574.007386] Lustre: Skipped 1 previous similar message [15579.111473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15579.118094] Lustre: Skipped 1 previous similar message [15580.127890] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15580.272913] Lustre: server umount lustre-MDT0000 complete [15584.524052] LustreError: 523022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876376 with bad export cookie 8964366307688749296 [15584.542601] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15584.545420] LustreError: 523022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15585.193710] Lustre: server umount lustre-MDT0001 complete [15612.035154] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [15621.512786] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [15655.465161] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15376 sec ========= 20:20:45 (1787876445) [15657.518767] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:20:47 (1787876447) === [15661.680642] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:20:51 (1787876451) === [15707.708871] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [15710.115126] Key type lgssc unregistered [15710.375894] LNet: 531650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15710.388271] LNetError: 531650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15710.403505] LNet: Removed LNI 192.168.203.116@tcp [15711.210968] Key type .llcrypt unregistered [15711.213122] Key type ._llcrypt unregistered