[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494580105 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003082] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007953] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010146] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013061] SELinux: Initializing. [ 0.014081] *** VALIDATE selinux *** [ 0.022533] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027789] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030030] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031117] *** VALIDATE tmpfs *** [ 0.033113] *** VALIDATE proc *** [ 0.034292] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.037187] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.042012] Speculative Store Bypass: Vulnerable [ 0.045491] debug: unmapping init [mem 0xffffffff9f259000-0xffffffff9f260fff] [ 0.047185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048719] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050019] ... bit width: 48 [ 0.051097] ... generic registers: 4 [ 0.052018] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056293] rcu: Hierarchical SRCU implementation. [ 0.058574] smp: Bringing up secondary CPUs ... [ 0.059638] x86: Booting SMP configuration: [ 0.060030] .... node #0, CPUs: #1 #2 #3 [ 0.063342] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105792] node 0 deferred pages initialised in 38ms [ 0.110173] devtmpfs: initialized [ 0.112272] x86/mm: Memory block size: 128MB [ 0.114970] gcov: version magic: 0x41383552 [ 0.118161] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122141] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.124351] pinctrl core: initialized pinctrl subsystem [ 0.126176] [ 0.126872] ************************************************************* [ 0.129011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131013] ** ** [ 0.134014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.136012] ** ** [ 0.139014] ** This means that this kernel is built to expose internal ** [ 0.141011] ** IOMMU data structures, which may compromise security on ** [ 0.144015] ** your system. ** [ 0.146012] ** ** [ 0.149015] ** If you see this message and you are not debugging the ** [ 0.152012] ** kernel, report this immediately to your vendor! ** [ 0.154012] ** ** [ 0.157011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159014] ************************************************************* [ 0.162832] NET: Registered protocol family 16 [ 0.164521] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178034] cpuidle: using governor menu [ 0.180753] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183541] PCI: Using configuration type 1 for base access [ 0.186138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195123] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199028] cryptd: max_cpu_qlen set to 1000 [ 0.201230] ACPI: Added _OSI(Module Device) [ 0.203016] ACPI: Added _OSI(Processor Device) [ 0.204013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211292] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.217375] ACPI: Interpreter enabled [ 0.219079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221029] ACPI: Using IOAPIC for interrupt routing [ 0.223138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226407] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239109] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.247090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.252341] acpiphp: Slot [2] registered [ 0.254132] acpiphp: Slot [5] registered [ 0.255166] acpiphp: Slot [6] registered [ 0.256093] acpiphp: Slot [7] registered [ 0.258125] acpiphp: Slot [8] registered [ 0.259142] acpiphp: Slot [9] registered [ 0.260143] acpiphp: Slot [10] registered [ 0.262133] acpiphp: Slot [3] registered [ 0.264096] acpiphp: Slot [4] registered [ 0.265152] acpiphp: Slot [11] registered [ 0.267111] acpiphp: Slot [12] registered [ 0.269111] acpiphp: Slot [13] registered [ 0.270121] acpiphp: Slot [14] registered [ 0.271101] acpiphp: Slot [15] registered [ 0.272128] acpiphp: Slot [16] registered [ 0.274095] acpiphp: Slot [17] registered [ 0.276134] acpiphp: Slot [18] registered [ 0.277099] acpiphp: Slot [19] registered [ 0.278121] acpiphp: Slot [20] registered [ 0.280168] acpiphp: Slot [21] registered [ 0.282084] acpiphp: Slot [22] registered [ 0.283122] acpiphp: Slot [23] registered [ 0.285119] acpiphp: Slot [24] registered [ 0.286097] acpiphp: Slot [25] registered [ 0.287146] acpiphp: Slot [26] registered [ 0.288085] acpiphp: Slot [27] registered [ 0.290121] acpiphp: Slot [28] registered [ 0.291153] acpiphp: Slot [29] registered [ 0.293241] acpiphp: Slot [30] registered [ 0.295115] acpiphp: Slot [31] registered [ 0.296062] PCI host bridge to bus 0000:00 [ 0.298023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.303028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.310028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.312226] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.315066] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.318437] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328812] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.333366] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.335018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.337034] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.340021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.343615] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.346912] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354838] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.374860] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.379011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384800] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.391016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.397016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.420019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.432802] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.443018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.449020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.480925] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.491015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.497019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.512021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.521134] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.528017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.536018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.574021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.587229] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.595018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.607027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.628029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.640185] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.648020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.665018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.694017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.711141] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.716220] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.720223] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.724529] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728955] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.736540] iommu: Default domain type: Passthrough [ 0.739582] SCSI subsystem initialized [ 0.741174] ACPI: bus type USB registered [ 0.742172] usbcore: registered new interface driver usbfs [ 0.745101] usbcore: registered new interface driver hub [ 0.747103] usbcore: registered new device driver usb [ 0.749207] pps_core: LinuxPPS API ver. 1 registered [ 0.751014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.754090] PTP clock support registered [ 0.756195] EDAC MC: Ver: 3.0.0 [ 0.762189] PCI: Using ACPI for IRQ routing [ 0.764933] NetLabel: Initializing [ 0.766013] NetLabel: domain hash size = 128 [ 0.768014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.770090] NetLabel: unlabeled traffic allowed by default [ 0.773137] vgaarb: loaded [ 0.774289] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.777019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.783919] clocksource: Switched to clocksource kvm-clock [ 0.898178] VFS: Disk quotas dquot_6.6.0 [ 0.900212] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.902797] *** VALIDATE ramfs *** [ 0.903895] *** VALIDATE hugetlbfs *** [ 0.905569] pnp: PnP ACPI init [ 0.908196] pnp: PnP ACPI: found 6 devices [ 0.927293] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.932507] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935058] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.937471] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.939974] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.942699] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.945083] NET: Registered protocol family 2 [ 0.947400] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.953620] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.957376] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.963100] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.967352] TCP: Hash tables configured (established 65536 bind 65536) [ 0.970724] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.974201] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.977435] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.980795] NET: Registered protocol family 1 [ 0.983780] RPC: Registered named UNIX socket transport module. [ 0.986450] RPC: Registered udp transport module. [ 0.988615] RPC: Registered tcp transport module. [ 0.990692] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.993493] NET: Registered protocol family 44 [ 0.995380] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.997795] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.000301] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.003398] PCI: CLS 0 bytes, default 64 [ 1.005295] Unpacking initramfs... [ 2.402749] debug: unmapping init [mem 0xffff9ee8fcc54000-0xffff9ee8fffbffff] [ 2.406699] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.409251] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.412460] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.923667] Initialise system trusted keyrings [ 2.925726] Key type blacklist registered [ 2.927716] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.937207] zbud: loaded [ 2.941910] *** VALIDATE nfs *** [ 2.943312] *** VALIDATE nfs4 *** [ 2.945800] pstore: using deflate compression [ 2.950598] Platform Keyring initialized [ 3.061554] NET: Registered protocol family 38 [ 3.066404] Key type asymmetric registered [ 3.068282] Asymmetric key parser 'x509' registered [ 3.070104] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.073678] io scheduler mq-deadline registered [ 3.075016] io scheduler kyber registered [ 3.076155] io scheduler bfq registered [ 3.078583] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.081958] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.084892] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.087768] ACPI: Power Button [PWRF] [ 3.093863] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.101117] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.114045] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.120506] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.135897] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.164820] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.196754] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.203350] Non-volatile memory driver v1.3 [ 3.205202] Linux agpgart interface v0.103 [ 3.236487] virtio_blk virtio1: [vda] 146752 512-byte logical blocks (75.1 MB/71.7 MiB) [ 3.239783] vda: detected capacity change from 0 to 75137024 [ 3.256307] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.259360] vdb: detected capacity change from 0 to 1073741824 [ 3.276790] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.280074] vdc: detected capacity change from 0 to 2621440000 [ 3.295687] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.301572] vdd: detected capacity change from 0 to 2621440000 [ 3.319449] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.322780] vde: detected capacity change from 0 to 4294967296 [ 3.337175] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.340511] vdf: detected capacity change from 0 to 4294967296 [ 3.347107] libphy: Fixed MDIO Bus: probed [ 3.352357] usbcore: registered new interface driver usbserial_generic [ 3.355081] usbserial: USB Serial support registered for generic [ 3.357488] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.362340] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.364384] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.367218] mousedev: PS/2 mouse device common for all mice [ 3.370625] rtc_cmos 00:05: RTC can wake from S4 [ 3.373974] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.379086] rtc_cmos 00:05: registered as rtc0 [ 3.381780] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.382609] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.385341] intel_pstate: CPU model not supported [ 3.392464] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.393530] hid: raw HID events driver (C) Jiri Kosina [ 3.399630] usbcore: registered new interface driver usbhid [ 3.401988] usbhid: USB HID core driver [ 3.404049] drop_monitor: Initializing network drop monitor service [ 3.407231] Initializing XFRM netlink socket [ 3.409728] NET: Registered protocol family 10 [ 3.413204] Segment Routing with IPv6 [ 3.415126] NET: Registered protocol family 17 [ 3.417986] mpls_gso: MPLS GSO support [ 3.426292] RAS: Correctable Errors collector initialized. [ 3.428977] AVX version of gcm_enc/dec engaged. [ 3.430809] AES CTR mode by8 optimization enabled [ 3.519649] sched_clock: Marking stable (3519625607, 0)->(4466864528, -947238921) [ 3.523114] registered taskstats version 1 [ 3.525355] Loading compiled-in X.509 certificates [ 3.527479] zswap: loaded using pool lzo/zbud [ 3.554972] Key type big_key registered [ 3.569060] Key type encrypted registered [ 3.570893] ima: No TPM chip found, activating TPM-bypass! [ 3.572960] ima: Allocated hash algorithm: sha1 [ 3.574726] ima: No architecture policies found [ 3.576414] evm: Initialising EVM extended attributes: [ 3.578510] evm: security.selinux [ 3.579785] evm: security.ima [ 3.580922] evm: security.capability [ 3.582018] evm: HMAC attrs: 0x1 [ 3.585125] rtc_cmos 00:05: setting system clock to 2026-09-08 03:33:26 UTC (1788838406) [ 3.591339] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 3.594735] debug: unmapping init [mem 0xffffffff9ef82000-0xffffffff9f258fff] [ 3.604897] Write protecting the kernel read-only data: 28672k [ 3.608142] debug: unmapping init [mem 0xffffffff9d603000-0xffffffff9d7fffff] [ 3.611291] debug: unmapping init [mem 0xffffffff9df14000-0xffffffff9dffffff] [ 3.644811] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.654306] systemd[1]: Detected virtualization kvm. [ 3.656404] systemd[1]: Detected architecture x86-64. [ 3.658320] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.687825] systemd[1]: No hostname configured. [ 3.689718] systemd[1]: Set hostname to . [ 3.691631] random: systemd: uninitialized urandom read (16 bytes read) [ 3.694359] systemd[1]: Initializing machine ID from random generator. [ 3.736459] random: ln: uninitialized urandom read (6 bytes read) [ 3.818536] random: systemd: uninitialized urandom read (16 bytes read) [ 3.821213] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.827925] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.833289] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.506900] device-mapper: uevent: version 1.0.3 [ 4.509526] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.291640] random: fast init done [ 5.319994] virtio_net virtio0 ens2: renamed from eth0 [ 5.374616] scsi host0: ata_piix [ 5.411765] scsi host1: ata_piix [ 5.413354] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.416650] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.736225] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.088509] random: crng init done [ 10.090070] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.540906] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.704073] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.997235] SELinux: Disabled at runtime. [ 12.065946] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.075194] systemd[1]: Detected virtualization kvm. [ 12.077250] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.660529] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.663404] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.669121] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.672275] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.675312] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.683017] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.687515] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ 12.868034] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.092959] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.421883] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.463301] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.547914] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.569318] EDAC sbridge: Ver: 1.1.2 [ 15.106208] Key type dns_resolver registered [ 15.406383] NFS: Registering the id_resolver key type [ 15.409709] Key type id_resolver registered [ 15.411470] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg415-server login: [ 41.947227] libcfs: loading out-of-tree module taints kernel. [ 41.967761] Key type ._llcrypt registered [ 41.969039] Key type .llcrypt registered [ 42.020673] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 51.277881] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 52.085308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.096372] alg: No test for adler32 (adler32-zlib) [ 53.926766] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 55.134454] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 57.024227] Key type lgssc registered [ 58.927416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.054055] hrtimer: interrupt took 6511267 ns [ 78.494969] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.452134] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 137.887783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.916271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.174632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.197763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.281988] Lustre: lustre-MDT0000: new disk, initializing [ 139.396103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.410387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.123773] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.711896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.916374] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.006896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.014236] Lustre: Skipped 1 previous similar message [ 158.138312] Lustre: lustre-MDT0001: new disk, initializing [ 158.208498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.252333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.266850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.055026] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.054187] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 178.415573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 178.711497] Lustre: lustre-OST0000: new disk, initializing [ 178.718710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 178.744583] Lustre: 8419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 178.833764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.101761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 182.108587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 182.181383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 185.824892] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.104851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.251451] Lustre: lustre-OST0001: new disk, initializing [ 200.256936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 200.261573] Lustre: 9491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.337882] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.924715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.940790] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.022715] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 208.147849] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.501965] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.646970] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.706382] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.942163] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing yml_node [ 242.873472] Lustre: DEBUG MARKER: Client: 2.17.57.117 [ 245.924943] Lustre: DEBUG MARKER: MDS: 2.17.57.117 [ 249.423986] Lustre: DEBUG MARKER: OSS: 2.17.57.117 [ 251.243769] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 23:37:31 EDT 2026 [ 271.257652] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 273.143171] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 289.249163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.250350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.269445] Lustre: Skipped 3 previous similar messages [ 289.279638] Lustre: Skipped 3 previous similar messages [ 294.369167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.374022] Lustre: Skipped 3 previous similar messages [ 294.958956] Lustre: server umount lustre-MDT0000 complete [ 298.622749] LustreError: 10368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838701 with bad export cookie 4661218207522808184 [ 298.623944] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.636547] LustreError: 10368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 299.496782] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.504250] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.514940] Lustre: Skipped 1 previous similar message [ 299.530118] Lustre: Skipped 1 previous similar message [ 303.521462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 303.530403] Lustre: Skipped 1 previous similar message [ 305.126445] Lustre: server umount lustre-MDT0001 complete [ 308.655416] Lustre: server umount lustre-OST0000 complete [ 312.291484] Lustre: server umount lustre-OST0001 complete [ 320.938694] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 333.115910] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 369.918465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.139909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 370.164566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 370.249191] Lustre: lustre-MDT0000: new disk, initializing [ 370.393048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.410361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 375.067554] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 386.538692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.698593] Lustre: 15049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 386.773358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 386.780579] Lustre: Skipped 1 previous similar message [ 386.870804] Lustre: lustre-MDT0001: new disk, initializing [ 386.993340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 387.030288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 387.044797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.953894] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 403.018065] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 409.213335] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 416.151206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 416.457351] Lustre: lustre-OST0000: new disk, initializing [ 416.460753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 416.468213] Lustre: 17020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 416.576511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.606956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 417.630358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 417.740216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 422.810590] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 432.136194] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 437.138986] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 437.379150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 442.558474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 442.849772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 448.489836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 448.493778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.501721] LustreError: Skipped 1 previous similar message [ 448.505443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 448.536143] Lustre: Skipped 1 previous similar message [ 458.727538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 458.744909] Lustre: Skipped 4 previous similar messages [ 459.233232] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 459.425494] Lustre: server umount lustre-OST0000 complete [ 463.842569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 463.859939] Lustre: Skipped 1 previous similar message [ 463.866808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 469.669669] Lustre: server umount lustre-MDT0000 complete [ 473.365520] LustreError: 15042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838876 with bad export cookie 4661218207522809717 [ 473.372148] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 473.378794] LustreError: 15042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 473.801526] Lustre: server umount lustre-MDT0001 complete [ 478.188297] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:41:18 (1788838878) [ 480.354939] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 482.321114] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:41:23 (1788838883) [ 489.505980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.096447] LustreError: 19429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.122113] LustreError: 19429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 490.190245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.435862] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 495.594507] LustreError: 19429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.705508] LustreError: 19430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.571115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.493608] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 517.752957] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 524.356347] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 532.517961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 540.111966] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 548.940496] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 560.611962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 560.614369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.640389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 560.646446] Lustre: Skipped 5 previous similar messages [ 566.752477] Lustre: server umount lustre-OST0000 complete [ 570.856315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.876225] Lustre: Skipped 2 previous similar messages [ 576.847144] Lustre: server umount lustre-MDT0000 complete [ 581.016703] LustreError: 21274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838983 with bad export cookie 4661218207522810599 [ 581.026458] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.035978] LustreError: 21274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 581.442040] Lustre: server umount lustre-MDT0001 complete [ 595.381960] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 598.866222] Key type lgssc unregistered [ 599.164598] LNet: 22964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.171575] LNetError: 22964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 599.205824] LNet: Removed LNI 192.168.204.115@tcp [ 600.240378] Key type .llcrypt unregistered [ 600.243867] Key type ._llcrypt unregistered [ 624.657463] Key type ._llcrypt registered [ 624.660250] Key type .llcrypt registered [ 624.847932] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 626.402419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 626.419234] alg: No test for adler32 (adler32-zlib) [ 627.533469] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 627.862317] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 629.568251] Key type lgssc registered [ 630.758914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 639.462050] loop: module loaded [ 639.566437] loop0: detected capacity change from 0 to 204800000 [ 639.809495] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.829333] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.117575] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 646.879372] loop0: detected capacity change from 0 to 204800000 [ 647.071521] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 647.088053] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 647.105055] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 652.644764] loop0: detected capacity change from 0 to 204800000 [ 658.420141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 658.446886] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.832293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 659.955872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 660.102617] Lustre: lustre-MDT0000: new disk, initializing [ 660.239395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.270232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 665.519761] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 675.877328] loop1: detected capacity change from 0 to 204800000 [ 682.119199] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 682.433746] Lustre: lustre-OST07c6: new disk, initializing [ 682.439819] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 682.450361] Lustre: Skipped 1 previous similar message [ 682.456574] Lustre: 26151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 682.554121] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 689.730786] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 689.765212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 689.777535] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 689.819262] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 694.794806] Lustre: 25138:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 694.808287] Lustre: 25138:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 694.826961] Lustre: 25138:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 694.841783] Lustre: 25138:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 694.859442] Lustre: 25138:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 694.868560] Lustre: 25138:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 712.281638] Lustre: 26668:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 712.294562] Lustre: 26668:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 712.304717] Lustre: 26668:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 712.314068] Lustre: 26668:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 712.323816] Lustre: 26668:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 712.328826] Lustre: 26668:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 717.237281] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 717.249063] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 717.255838] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 717.262610] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.274074] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.284892] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.293697] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 717.305884] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.313556] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 717.321967] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.331978] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 717.339662] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.596758] Lustre: 26668:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 720.602490] Lustre: 26668:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 720.612420] Lustre: 26668:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 720.619731] Lustre: 26668:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.626130] Lustre: 26668:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.631899] Lustre: 26668:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.640558] Lustre: 26668:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 720.648077] Lustre: 26668:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.658148] Lustre: 26668:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 720.663996] Lustre: 26668:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.682331] Lustre: 26668:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.692455] Lustre: 26668:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.738591] Lustre: 25138:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 725.748475] Lustre: 25138:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.752474] Lustre: 25138:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.756139] Lustre: 25138:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.766363] Lustre: 25138:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.790198] Lustre: 25138:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.809739] Lustre: 25138:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 725.827183] Lustre: 25138:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.836362] Lustre: 25138:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.842874] Lustre: 25138:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.851172] Lustre: 25138:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.860414] Lustre: 25138:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 737.298580] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 737.308300] Lustre: 25139:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 737.317767] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 737.329807] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.342585] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 737.349359] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.362091] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 737.374158] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.379816] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 737.386267] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.398522] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 737.411567] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 786.921936] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.938932] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 786.944991] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 792.036061] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 797.159864] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 800.225151] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 800.398599] Lustre: server umount lustre-OST07c6 complete [ 811.489750] Lustre: server umount lustre-MDT0000 complete [ 831.162327] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 835.918979] Key type lgssc unregistered [ 836.354086] LNet: 28712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.367393] LNetError: 28712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.390778] LNet: Removed LNI 192.168.204.115@tcp [ 837.424303] Key type .llcrypt unregistered [ 837.428085] Key type ._llcrypt unregistered [ 859.474896] Key type ._llcrypt registered [ 859.478006] Key type .llcrypt registered [ 859.597373] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 860.925173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 860.947429] alg: No test for adler32 (adler32-zlib) [ 861.998551] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 862.212936] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 863.872374] Key type lgssc registered [ 865.494247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 876.033873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 876.067232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.964020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.949076] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 887.284700] LustreError: 29976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.308905] LustreError: 29976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 892.387523] LustreError: 29977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.908989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.280146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 898.706350] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 906.909950] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 912.269935] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 919.351086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 919.718746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.809341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 926.328935] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 934.483209] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 951.777757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.784353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.802542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 952.290897] Lustre: server umount lustre-OST0000 complete [ 958.956408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.996311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 958.999052] Lustre: Skipped 1 previous similar message [ 964.072045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.079459] Lustre: Skipped 1 previous similar message [ 966.211872] Lustre: server umount lustre-MDT0000 complete [ 968.161819] LustreError: 29977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.192680] LustreError: 29977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 970.141721] LustreError: 29957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839373 with bad export cookie 14647444850610243115 [ 970.142511] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.154206] LustreError: 29957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 970.519507] Lustre: server umount lustre-MDT0001 complete [ 983.522274] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 985.963459] Key type lgssc unregistered [ 986.239061] LNet: 33716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.250842] LNetError: 33716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.315858] LNet: Removed LNI 192.168.204.115@tcp [ 988.401188] Key type .llcrypt unregistered [ 988.407559] Key type ._llcrypt unregistered [ 1013.678601] Key type ._llcrypt registered [ 1013.680700] Key type .llcrypt registered [ 1013.904410] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1015.333040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.377940] alg: No test for adler32 (adler32-zlib) [ 1016.433455] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1016.607839] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1018.384459] Key type lgssc registered [ 1020.251589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.117662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.147831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.622091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.114221] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1040.868574] LustreError: 34986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.885646] LustreError: 34986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1045.094215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.544304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1050.745778] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1058.281311] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1064.404184] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1070.649620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1071.012160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.146058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1077.070527] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1084.983532] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1100.782222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1100.791045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.807089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.811749] Lustre: Skipped 1 previous similar message [ 1100.842421] Lustre: Skipped 1 previous similar message [ 1104.200506] Lustre: server umount lustre-OST0000 complete [ 1111.009247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.024740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.029376] Lustre: Skipped 1 previous similar message [ 1113.806671] Lustre: server umount lustre-MDT0000 complete [ 1116.146717] LustreError: 35714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.186699] LustreError: 35714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1118.726770] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839521 with bad export cookie 9585120960114146329 [ 1118.739988] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1119.091494] Lustre: server umount lustre-MDT0001 complete [ 1129.957420] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1133.839455] Key type lgssc unregistered [ 1134.179619] LNet: 38540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.189759] LNetError: 38540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.216592] LNet: Removed LNI 192.168.204.115@tcp [ 1135.384884] Key type .llcrypt unregistered [ 1135.387572] Key type ._llcrypt unregistered [ 1164.166216] Key type ._llcrypt registered [ 1164.167893] Key type .llcrypt registered [ 1164.297216] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1165.304090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.358262] alg: No test for adler32 (adler32-zlib) [ 1166.407089] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1166.666260] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1168.320170] Key type lgssc registered [ 1169.536901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1177.715460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1177.745852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.379830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.230357] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1189.860545] LustreError: 39955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.902498] LustreError: 39955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1194.979738] LustreError: 39954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.462927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.065243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1202.127571] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1211.162912] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1218.259685] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1225.961169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1226.378339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.363957] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1242.702498] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1249.984024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1259.614673] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1260.528470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.535594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1260.548251] Lustre: Skipped 2 previous similar messages [ 1260.559120] Lustre: Skipped 2 previous similar messages [ 1261.802526] Lustre: server umount lustre-MDT0000 complete [ 1264.707142] LustreError: 40889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.562540] LustreError: 39935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839668 with bad export cookie 3733642747274108381 [ 1265.574544] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1265.579245] LustreError: 39935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1265.641496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.660115] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1269.815685] LustreError: 40889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.829923] LustreError: 40889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1269.841670] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 1271.961434] Lustre: server umount lustre-MDT0001 complete [ 1293.760565] Lustre: server umount lustre-OST0000 complete [ 1312.475174] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1314.688155] Key type lgssc unregistered [ 1314.899742] LNet: 43835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.905719] LNetError: 43835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.922944] LNet: Removed LNI 192.168.204.115@tcp [ 1315.595944] Key type .llcrypt unregistered [ 1315.598276] Key type ._llcrypt unregistered [ 1341.419701] Key type ._llcrypt registered [ 1341.422509] Key type .llcrypt registered [ 1341.584387] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1343.110045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1343.238857] alg: No test for adler32 (adler32-zlib) [ 1344.436458] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1344.659381] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1346.352866] Key type lgssc registered [ 1347.791676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1357.641262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1357.707732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.276175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.477310] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1369.570119] LustreError: 45249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.604521] LustreError: 45249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1373.283381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.793237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1379.006820] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1388.126488] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1395.207608] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1402.294327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1402.718071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.819802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1410.121333] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1418.588629] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1426.018924] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1429.475112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.480796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1429.494840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.110898] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1431.133482] Lustre: Skipped 1 previous similar message [ 1434.594475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1439.721881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1439.732406] Lustre: Skipped 2 previous similar messages [ 1440.225180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1440.554048] Lustre: server umount lustre-OST0000 complete [ 1460.195083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.200609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.205905] Lustre: Skipped 2 previous similar messages [ 1460.216906] Lustre: Skipped 2 previous similar messages [ 1466.166271] Lustre: server umount lustre-MDT0000 complete [ 1469.412169] LustreError: 45249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.422586] LustreError: 45249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1470.586286] LustreError: 45229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839873 with bad export cookie 2652263768822445812 [ 1470.602027] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1470.605690] LustreError: 45229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1476.967615] Lustre: server umount lustre-MDT0001 complete [ 1488.245583] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1491.309405] Key type lgssc unregistered [ 1491.617588] LNet: 48958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.627865] LNetError: 48958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.644985] LNet: Removed LNI 192.168.204.115@tcp [ 1492.356121] Key type .llcrypt unregistered [ 1492.359206] Key type ._llcrypt unregistered [ 1517.224907] Key type ._llcrypt registered [ 1517.227785] Key type .llcrypt registered [ 1517.363209] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1518.461805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1518.557667] alg: No test for adler32 (adler32-zlib) [ 1519.681723] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1520.022472] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1521.712267] Key type lgssc registered [ 1522.811433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1530.934858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1530.986307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.325358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.626433] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1542.655864] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.673818] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1545.153568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.569290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1549.576463] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1559.605920] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1566.279868] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1574.254547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.548930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.665932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1581.657501] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1590.028810] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1596.947349] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1600.934238] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1613.435368] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 0h00m13s [ 1615.274486] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1618.504418] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1619.577063] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1621.988780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.995722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1622.021063] Lustre: Skipped 1 previous similar message [ 1622.030380] Lustre: Skipped 1 previous similar message [ 1624.008460] Lustre: server umount lustre-MDT0000 complete [ 1624.632091] LustreError: 50368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1627.495921] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1629.173849] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.198703] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1634.947041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.131897] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1635.415546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1635.475046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1636.462120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1640.162785] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1643.961583] Lustre: DEBUG MARKER: Wait for df (36664) ... [ 1645.783602] Lustre: DEBUG MARKER: done [ 1650.658938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1650.675238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.699549] Lustre: Skipped 1 previous similar message [ 1650.706434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1650.714521] Lustre: Skipped 4 previous similar messages [ 1652.201044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.901243] Lustre: server umount lustre-OST0000 complete [ 1662.435382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.435882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1662.453164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.457199] Lustre: Skipped 2 previous similar messages [ 1673.704088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.710404] Lustre: Skipped 5 previous similar messages [ 1675.232838] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1675.421876] Lustre: server umount lustre-MDT0000 complete [ 1678.679966] LustreError: 50354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840081 with bad export cookie 3607635263213704512 [ 1678.701650] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1678.707514] LustreError: 50354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1678.985440] Lustre: server umount lustre-MDT0001 complete [ 1691.312964] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1694.733308] Key type lgssc unregistered [ 1695.019962] LNet: 55316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.024565] LNetError: 55316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.043823] LNet: Removed LNI 192.168.204.115@tcp [ 1695.900175] Key type .llcrypt unregistered [ 1695.903545] Key type ._llcrypt unregistered [ 1739.027071] Key type ._llcrypt registered [ 1739.032507] Key type .llcrypt registered [ 1739.174043] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1740.191426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1740.221858] alg: No test for adler32 (adler32-zlib) [ 1741.245537] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1741.618246] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1743.370577] Key type lgssc registered [ 1745.018112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1753.798823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1753.824197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.160530] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1755.205841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1755.464419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.178813] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1770.618413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.712645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1770.741323] Lustre: 57524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1771.076466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1776.421522] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1784.720148] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1791.184754] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1798.385427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1798.656681] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1798.917852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.531528] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1808.562972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1815.735340] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1822.396624] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 0h03m42s [ 1824.220388] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1833.082616] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1835.789710] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1836.075861] LustreError: 60056:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1856.625122] Lustre: lustre-MDT0000: Client 3714bc60-b417-4996-a84c-b491960fe11c (at 192.168.204.15@tcp) reconnecting [ 1876.128135] LustreError: 60056:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1878.654818] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1881.827697] Lustre: DEBUG MARKER: done [ 1890.790725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.798380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.803773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.827919] Lustre: Skipped 1 previous similar message [ 1894.960547] Lustre: server umount lustre-OST0000 complete [ 1901.029380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.032991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.046760] Lustre: Skipped 1 previous similar message [ 1901.060602] Lustre: Skipped 2 previous similar messages [ 1902.116754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1902.125854] Lustre: Skipped 1 previous similar message [ 1905.764579] Lustre: server umount lustre-MDT0000 complete [ 1909.692877] LustreError: 57517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840312 with bad export cookie 10105184367568308814 [ 1909.700393] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.704059] LustreError: 57517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1910.100936] Lustre: server umount lustre-MDT0001 complete [ 1921.759394] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1924.811653] Key type lgssc unregistered [ 1925.128899] LNet: 61855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.148152] LNetError: 61855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.190295] LNet: Removed LNI 192.168.204.115@tcp [ 1927.045508] Key type .llcrypt unregistered [ 1927.048827] Key type ._llcrypt unregistered [ 1980.134356] Key type ._llcrypt registered [ 1980.136856] Key type .llcrypt registered [ 1980.246959] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1982.082175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1982.126075] alg: No test for adler32 (adler32-zlib) [ 1983.254630] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 1983.788089] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1985.514838] Key type lgssc registered [ 1986.909235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1992.013479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1992.040725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.466635] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1993.501437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1993.705061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.322686] Lustre: Failing over lustre-MDT0000 [ 1995.731928] Lustre: server umount lustre-MDT0000 complete [ 2004.148324] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:06:44 (1788840404) [ 2013.593946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.347812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2020.758156] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2032.694439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.799798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2032.843878] Lustre: 64654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2033.172793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2038.154596] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2048.369677] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2055.219524] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2063.730168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2063.894079] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2064.122380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.467734] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2073.329247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2081.729957] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2086.993365] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2099.173260] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.178155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2099.194314] Lustre: Skipped 1 previous similar message [ 2099.218277] Lustre: Skipped 2 previous similar messages [ 2101.429607] Lustre: server umount lustre-MDT0000 complete [ 2105.545759] LustreError: 64649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840508 with bad export cookie 13670254787102162593 [ 2105.579900] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2105.957531] Lustre: server umount lustre-MDT0001 complete [ 2109.678891] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2115.085250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2125.247225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.792580] Lustre: 63576:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840512/real 1788840512] req@00000000c63ae05c x1875735096634368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788840528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.859617] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.873311] Lustre: Skipped 1 previous similar message [ 2126.816600] Lustre: 63577:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840513/real 1788840513] req@0000000060c1e3b6 x1875735096634496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788840529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.913212] Lustre: 63578:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840517/real 1788840517] req@00000000a64c84fa x1875735096634624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788840533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2131.043838] LustreError: 63575:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000200ec4bd x1875735096635520/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2131.469831] LustreError: 66506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.498725] LustreError: 66506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2131.609296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.744263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2135.587724] LustreError: 66507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.588088] Lustre: 63577:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840523/real 1788840523] req@00000000f5ba5dfb x1875735096635008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788840539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.611556] Lustre: 63577:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2136.613865] LustreError: 68427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.657739] LustreError: 68427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2137.086142] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2141.666783] LustreError: 68428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.687291] LustreError: 68428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2142.688361] Lustre: 63577:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840528/real 1788840528] req@00000000ecf87ed2 x1875735096635136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788840544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2146.801089] LustreError: 66506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.826701] LustreError: 66506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2148.160590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.510244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2152.922587] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2153.974759] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2155.047443] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2161.522687] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2169.010562] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.041340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2179.056755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.076537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2180.591622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.481064] Lustre: server umount lustre-MDT0000 complete [ 2185.702820] LustreError: 69971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.726887] LustreError: 69971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2187.617759] LustreError: 64647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840590 with bad export cookie 13670254787102164868 [ 2187.628023] LustreError: 64647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2187.645813] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2188.129796] Lustre: server umount lustre-MDT0001 complete [ 2196.758104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2206.178177] Lustre: 63577:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840593/real 1788840593] req@000000008d1d1793 x1875735096664448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788840609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2206.242208] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.258512] Lustre: Skipped 2 previous similar messages [ 2208.371157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.874652] LustreError: 69163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.018511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.102650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2218.403309] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2227.860937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.048381] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2233.844650] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2235.955488] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2240.693121] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2249.108620] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2258.914373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2258.936266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.972744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.978013] Lustre: Skipped 2 previous similar messages [ 2263.614306] Lustre: server umount lustre-MDT0000 complete [ 2266.594735] LustreError: 69163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.623509] LustreError: 69163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2267.492068] LustreError: 64649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840670 with bad export cookie 13670254787102166247 [ 2267.497112] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2267.511507] LustreError: 64649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2267.851718] Lustre: server umount lustre-MDT0001 complete [ 2273.860173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2277.541238] Lustre: DEBUG MARKER: files compared the same [ 2289.932248] Lustre: server umount lustre-OST0000 complete [ 2307.418290] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2310.906669] Key type lgssc unregistered [ 2311.237946] LNet: 74390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.246548] LNetError: 74390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.281511] LNet: Removed LNI 192.168.204.115@tcp [ 2312.308171] Key type .llcrypt unregistered [ 2312.311289] Key type ._llcrypt unregistered [ 2338.770442] Key type ._llcrypt registered [ 2338.772888] Key type .llcrypt registered [ 2338.861265] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2339.666507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2339.798567] alg: No test for adler32 (adler32-zlib) [ 2340.923831] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 2341.232195] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2342.936242] Key type lgssc registered [ 2344.700983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.817103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.834607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.554473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.216207] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 2367.969584] LustreError: 75806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.010241] LustreError: 75806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2373.093706] LustreError: 75807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.368543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.842584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2380.453978] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 2390.290354] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2397.377763] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2407.064512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2407.666666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.767948] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 2426.364705] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.128170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2440.685640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2440.703456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.745536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.723854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2442.730222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.737746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.776698] Lustre: Skipped 1 previous similar message [ 2446.196799] Lustre: server umount lustre-OST0000 complete [ 2451.939483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2451.946867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.963056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2458.083345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2458.099137] Lustre: Skipped 3 previous similar messages [ 2463.203406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2463.211699] Lustre: Skipped 1 previous similar message [ 2465.248148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2465.823277] Lustre: server umount lustre-MDT0000 complete [ 2467.297448] LustreError: 75801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.308397] LustreError: 75801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2470.637547] LustreError: 75788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840873 with bad export cookie 1487555828572981161 [ 2470.646060] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2470.657156] LustreError: 75788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2471.150491] Lustre: server umount lustre-MDT0001 complete [ 2485.796291] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2489.855793] Key type lgssc unregistered [ 2490.326873] LNet: 79364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2490.333229] LNetError: 79364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2490.345923] LNet: Removed LNI 192.168.204.115@tcp [ 2491.558817] Key type .llcrypt unregistered [ 2491.560895] Key type ._llcrypt unregistered [ 2520.871346] Key type ._llcrypt registered [ 2520.877506] Key type .llcrypt registered [ 2521.036168] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2522.862840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2522.925108] alg: No test for adler32 (adler32-zlib) [ 2523.954723] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 2524.161542] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2525.888216] Key type lgssc registered [ 2527.261875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2536.414325] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2536.465630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2552.161543] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2557.280303] LustreError: 80753:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 2583.904410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.760708] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2600.525524] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2609.305574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.561654] Lustre: *** cfs_fail_loc=706, val=0*** [ 2610.087720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.164870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2615.173109] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2615.276343] LustreError: 80778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.310642] LustreError: 80778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2618.354960] LustreError: 81838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.365932] LustreError: 81838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2623.460506] LustreError: 81838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.489969] LustreError: 81838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2624.988851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.308757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2630.798742] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2639.565716] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2645.806596] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2651.106264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2651.118768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.130322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2651.135209] Lustre: Skipped 1 previous similar message [ 2654.535493] Lustre: server umount lustre-OST0000 complete [ 2661.345980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.361811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2661.366845] Lustre: Skipped 2 previous similar messages [ 2664.930344] Lustre: server umount lustre-MDT0000 complete [ 2666.468744] LustreError: 81842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.491531] LustreError: 81842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2669.280679] LustreError: 80759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841072 with bad export cookie 5241159890928906083 [ 2669.282707] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2669.305394] LustreError: 80759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2669.644510] Lustre: server umount lustre-MDT0001 complete [ 2683.374516] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2687.214509] Key type lgssc unregistered [ 2687.566069] LNet: 84371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.572885] LNetError: 84371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2687.606933] LNet: Removed LNI 192.168.204.115@tcp [ 2688.674547] Key type .llcrypt unregistered [ 2688.681523] Key type ._llcrypt unregistered [ 2718.679844] Key type ._llcrypt registered [ 2718.682749] Key type .llcrypt registered [ 2718.798622] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2719.997237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2720.071157] alg: No test for adler32 (adler32-zlib) [ 2721.177682] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 2721.513912] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2723.354318] Key type lgssc registered [ 2724.552760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2733.136206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2733.159647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.577201] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2748.572575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.280918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2754.530326] LustreError: 86480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.557666] LustreError: 86480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2755.323654] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2759.651497] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.676018] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2764.550074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2764.774847] LustreError: 86484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.790551] LustreError: 86484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2764.916975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.417873] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.442143] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2772.190938] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2774.505218] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.532040] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2782.003395] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2782.019751] Lustre: MGS: Client f764a8de-4d17-4ea4-ac70-619a0428aec5 (at 0@lo) reconnecting [ 2782.041579] LustreError: 86495:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2782.325825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.419464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2787.333236] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2802.672201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.681083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.694285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2808.168714] Lustre: server umount lustre-OST0000 complete [ 2813.437745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2813.438259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.443716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2813.443725] Lustre: Skipped 2 previous similar messages [ 2813.501131] Lustre: Skipped 2 previous similar messages [ 2818.463893] Lustre: server umount lustre-MDT0000 complete [ 2818.530605] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.545830] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2823.038544] Lustre: server umount lustre-MDT0001 complete [ 2827.575558] Lustre: server umount MGS complete [ 2840.638265] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:20:41 (1788841241) [ 2873.995866] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 2889.723978] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2932.990276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.834864] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2947.827502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.120599] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2948.223307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2948.314689] Lustre: lustre-MDT0001: new disk, initializing [ 2948.382719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2948.424929] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2948.437554] Lustre: 93682:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2948.445610] LustreError: 93682:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2953.026250] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2963.778586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2964.175971] Lustre: lustre-OST0000: new disk, initializing [ 2964.185741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2964.198758] Lustre: 94449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2964.276191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2971.520357] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2973.288298] Lustre: 94828:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004fb55605 x1875735870480512/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788841392 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2982.487593] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2989.226253] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2989.248832] Lustre: MGS: Client a8d4bfa8-282e-40f6-a78d-5a587097d74c (at 0@lo) reconnecting [ 2989.308030] Lustre: 92951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2989.320614] LustreError: 93668:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2989.331312] LustreError: 93668:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2989.372310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2989.487409] Lustre: lustre-MDT0000: new disk, initializing [ 2989.659499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.697678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2989.716517] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2990.755572] Lustre: 95518:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ddf65d2a x1875735870487552/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788841409 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2994.707913] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2995.824268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2995.836581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2995.968963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3021.291985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.296579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.301929] Lustre: Skipped 1 previous similar message [ 3021.302597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.319863] Lustre: Skipped 2 previous similar messages [ 3025.387351] Lustre: server umount lustre-OST0000 complete [ 3031.522081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3031.525957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.551262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.571424] Lustre: Skipped 1 previous similar message [ 3031.598375] Lustre: Skipped 1 previous similar message [ 3036.642382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.648791] Lustre: Skipped 1 previous similar message [ 3045.345463] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3045.481549] Lustre: server umount lustre-MDT0000 complete [ 3046.896417] LustreError: 93654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.916351] LustreError: 93654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3055.462500] Lustre: server umount lustre-MDT0001 complete [ 3058.446333] Lustre: server umount MGS complete [ 3070.281774] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:24:31 (1788841471) [ 3088.875372] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3092.094185] Key type lgssc unregistered [ 3092.443654] LNet: 98015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3092.458138] LNetError: 98015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3092.481029] LNet: Removed LNI 192.168.204.115@tcp [ 3093.337470] Key type .llcrypt unregistered [ 3093.341125] Key type ._llcrypt unregistered [ 3113.864122] Key type ._llcrypt registered [ 3113.867928] Key type .llcrypt registered [ 3113.971409] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3115.196534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3115.220753] alg: No test for adler32 (adler32-zlib) [ 3116.279987] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 3116.510657] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3118.200227] Key type lgssc registered [ 3119.715414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3127.620140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3127.641028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3129.114661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3134.258313] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3134.444425] LustreError: 99236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.467606] LustreError: 99236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3138.540020] LustreError: 99236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.058876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.344151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3146.658275] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3155.535787] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3161.947686] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3169.157805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3169.631482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.747907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3176.022937] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3183.335895] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3198.435931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.439380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3198.439807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3198.454539] Lustre: Skipped 1 previous similar message [ 3198.481176] Lustre: Skipped 1 previous similar message [ 3201.888567] Lustre: server umount lustre-OST0000 complete [ 3208.675121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.688634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3208.701655] Lustre: Skipped 1 previous similar message [ 3208.717341] Lustre: Skipped 1 previous similar message [ 3211.756986] Lustre: server umount lustre-MDT0000 complete [ 3213.793473] LustreError: 99237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.813737] LustreError: 99237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3215.437614] LustreError: 101078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841618 with bad export cookie 2339556129385794626 [ 3215.443734] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3215.448533] LustreError: 101078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3215.797683] Lustre: server umount lustre-MDT0001 complete [ 3226.997842] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3230.062595] Key type lgssc unregistered [ 3230.401774] LNet: 102787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3230.414972] LNetError: 102787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3230.435149] LNet: Removed LNI 192.168.204.115@tcp [ 3231.205398] Key type .llcrypt unregistered [ 3231.207957] Key type ._llcrypt unregistered [ 3246.354827] Key type ._llcrypt registered [ 3246.359902] Key type .llcrypt registered [ 3246.461783] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3247.182626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3247.266371] alg: No test for adler32 (adler32-zlib) [ 3248.261718] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 3248.462951] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3250.120177] Key type lgssc registered [ 3251.077091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3256.533783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3256.549316] LustreError: 103868:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3256.552629] Lustre: Skipped 1 previous similar message [ 3256.561772] LustreError: 103864:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3256.570747] LustreError: 103868:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3256.579408] LustreError: 103868:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3256.592806] LustreError: 103868:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3256.604418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.176563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.208438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.445076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3269.385529] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3275.910987] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lsmod [ 3279.643224] LustreError: 105069:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3279.679816] LustreError: 105076:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3279.687475] LustreError: 105069:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4965 [ 3279.694320] LustreError: 105069:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3279.709867] LustreError: 105069:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3279.713461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3280.091337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3283.201097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3285.986231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3286.003201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.054215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.546879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.549495] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.559052] Lustre: Skipped 1 previous similar message [ 3288.596073] Lustre: Skipped 1 previous similar message [ 3290.660377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.665791] Lustre: Skipped 3 previous similar messages [ 3295.715479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3295.726944] Lustre: Skipped 2 previous similar messages [ 3298.272353] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3298.429224] Lustre: server umount lustre-MDT0000 complete [ 3300.842835] LustreError: 103902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3300.878660] LustreError: 103902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3301.690063] LustreError: 104344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841704 with bad export cookie 4657126757784110627 [ 3301.709075] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3301.714391] LustreError: 104344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3302.944304] LustreError: 104337:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006d4f0b92 x1875736423062912/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3303.118876] Lustre: server umount lustre-MDT0001 complete [ 3317.188497] Lustre: server umount lustre-OST0000 complete [ 3326.166163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.845661] LustreError: 106142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3326.950125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3331.994477] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3332.071776] LustreError: 106143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.165621] LustreError: 106142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3341.284139] LustreError: 106143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3341.932143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.362103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3347.264736] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3354.763583] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3361.515058] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3368.728538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.085257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3371.130197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3376.172193] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3384.826105] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3402.210442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3402.222425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.248248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.237278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3404.929668] Lustre: server umount lustre-OST0000 complete [ 3413.474394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.485760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.494590] Lustre: Skipped 2 previous similar messages [ 3413.503847] Lustre: Skipped 3 previous similar messages [ 3415.900729] Lustre: server umount lustre-MDT0000 complete [ 3418.598135] LustreError: 107134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.638058] LustreError: 107134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3420.368316] LustreError: 106124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841823 with bad export cookie 4657126757784111572 [ 3420.383764] LustreError: 106124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3420.384699] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3420.735338] Lustre: server umount lustre-MDT0001 complete [ 3431.972620] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3434.979711] Key type lgssc unregistered [ 3435.205166] LNet: 109693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3435.217506] LNetError: 109693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3435.244757] LNet: Removed LNI 192.168.204.115@tcp [ 3436.031843] Key type .llcrypt unregistered [ 3436.034824] Key type ._llcrypt unregistered [ 3460.184493] Key type ._llcrypt registered [ 3460.187784] Key type .llcrypt registered [ 3460.313376] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3461.464823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3461.481819] alg: No test for adler32 (adler32-zlib) [ 3462.578657] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 3462.803309] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3464.465297] Key type lgssc registered [ 3465.542550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3474.050656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3474.087085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.656462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.624980] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3485.162192] LustreError: 111105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3485.195627] LustreError: 111105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3490.276450] LustreError: 111106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.541692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.058185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3495.755347] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3504.721781] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3510.434157] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3517.104358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.472713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.562320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3523.793479] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3532.753753] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3541.597976] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3546.356060] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3556.326664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3556.327265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.332644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3556.341033] LustreError: Skipped 1 previous similar message [ 3556.372680] Lustre: Skipped 1 previous similar message [ 3558.215747] Lustre: server umount lustre-OST0000 complete [ 3562.999125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3563.013814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3563.034695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3563.043629] Lustre: Skipped 1 previous similar message [ 3566.563183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.563389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3566.590179] Lustre: Skipped 1 previous similar message [ 3571.683696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3571.693153] Lustre: Skipped 1 previous similar message [ 3576.288262] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3576.461825] Lustre: server umount lustre-MDT0000 complete [ 3576.803285] LustreError: 111106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.817919] LustreError: 111106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3580.136922] LustreError: 111085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841983 with bad export cookie 14541291802363671559 [ 3580.142825] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3580.154444] LustreError: 111085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3580.414029] Lustre: server umount lustre-MDT0001 complete [ 3591.967556] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3595.846493] Key type lgssc unregistered [ 3596.215995] LNet: 114748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3596.228637] LNetError: 114748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3596.252272] LNet: Removed LNI 192.168.204.115@tcp [ 3597.329166] Key type .llcrypt unregistered [ 3597.331352] Key type ._llcrypt unregistered [ 3617.260503] Key type ._llcrypt registered [ 3617.265754] Key type .llcrypt registered [ 3617.369393] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3618.417944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3618.457213] alg: No test for adler32 (adler32-zlib) [ 3619.555333] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 3619.742966] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3621.384180] Key type lgssc registered [ 3622.979807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3631.329340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3631.344291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.758689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.990396] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3642.356535] LustreError: 115967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.374528] LustreError: 115967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3647.473480] LustreError: 115966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.690696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.175749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3653.381295] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3661.682162] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3667.509626] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3675.340337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3675.742028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.824263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3682.713615] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3690.646902] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3702.762594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.770626] Lustre: Skipped 1 previous similar message [ 3702.774563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3702.778317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3702.794384] Lustre: Skipped 1 previous similar message [ 3706.467944] Lustre: server umount lustre-OST0000 complete [ 3713.000631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.001858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3713.028282] Lustre: Skipped 1 previous similar message [ 3713.045577] Lustre: Skipped 1 previous similar message [ 3717.627942] Lustre: server umount lustre-MDT0000 complete [ 3718.118174] LustreError: 115966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.149665] LustreError: 115966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3721.003713] LustreError: 115949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842123 with bad export cookie 7654280734006181621 [ 3721.017625] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3721.018651] LustreError: 115949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3721.325786] Lustre: server umount lustre-MDT0001 complete [ 3733.316937] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3735.895242] Key type lgssc unregistered [ 3736.174324] LNet: 119498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3736.185237] LNetError: 119498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3736.221355] LNet: Removed LNI 192.168.204.115@tcp [ 3737.067305] Key type .llcrypt unregistered [ 3737.070107] Key type ._llcrypt unregistered [ 3762.578440] Key type ._llcrypt registered [ 3762.580791] Key type .llcrypt registered [ 3762.755257] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3764.320822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3764.351287] alg: No test for adler32 (adler32-zlib) [ 3765.505711] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 3765.779490] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3767.520245] Key type lgssc registered [ 3768.876928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3777.337287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3777.351739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.976690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.990269] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3788.259678] LustreError: 120912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.279484] LustreError: 120912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3792.537399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.965057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3797.989414] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3805.878928] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3812.095767] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3818.529281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3818.835118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3823.989692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3824.748217] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3833.850976] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3842.877288] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3846.193173] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3846.202083] Lustre: Skipped 2 previous similar messages [ 3849.800092] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3849.818972] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3850.678570] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3850.684672] Lustre: Skipped 2 previous similar messages [ 3859.514090] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3859.522613] Lustre: Skipped 1 previous similar message [ 3864.245593] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3864.260841] Lustre: Skipped 2 previous similar messages [ 3869.761576] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.15@tcp 192.168.0.[2,10]@tcp [ 3874.504551] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.15/24@tcp in log lustre-MDT0000 [ 3874.519606] Lustre: Skipped 2 previous similar messages [ 3878.404551] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.15/24@tcp [ 3878.409310] Lustre: Skipped 1 previous similar message [ 3889.128056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3889.132568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.133534] LustreError: Skipped 2 previous similar messages [ 3889.154371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3891.242937] Lustre: server umount lustre-OST0000 complete [ 3898.859165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3898.860873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3898.869274] Lustre: Skipped 1 previous similar message [ 3901.652980] Lustre: server umount lustre-MDT0000 complete [ 3903.969720] LustreError: 124058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.986814] LustreError: 124058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3905.625343] LustreError: 120893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842308 with bad export cookie 14589955256861203818 [ 3905.636474] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3905.639155] LustreError: 120893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3915.575460] Lustre: server umount lustre-MDT0001 complete [ 3927.721176] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3931.317895] Key type lgssc unregistered [ 3931.965528] LNet: 125958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3931.976134] LNetError: 125958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3932.022916] LNet: Removed LNI 192.168.204.115@tcp [ 3933.330098] Key type .llcrypt unregistered [ 3933.335480] Key type ._llcrypt unregistered [ 3968.623379] Key type ._llcrypt registered [ 3968.625966] Key type .llcrypt registered [ 3968.765103] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3970.660581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3970.692958] alg: No test for adler32 (adler32-zlib) [ 3971.749986] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 3972.082522] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3973.768256] Key type lgssc registered [ 3975.363830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3984.010834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3984.024307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.485481] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3985.505300] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.15/24@tcp [ 3985.519884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3990.026779] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3995.620607] LustreError: 127520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.640839] LustreError: 127520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3998.007041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3998.345028] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3998.378724] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.15/24@tcp [ 3998.442751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4002.567308] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4010.504817] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4017.009922] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4024.971682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4025.259671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.426010] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4031.508603] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4039.924219] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4052.973673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4052.986232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.998895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4053.985978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4053.986385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4053.991483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.001038] Lustre: server umount lustre-OST0000 complete [ 4064.227708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.228949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4064.242596] Lustre: Skipped 1 previous similar message [ 4064.254884] Lustre: Skipped 2 previous similar messages [ 4068.132731] Lustre: server umount lustre-MDT0000 complete [ 4069.347846] LustreError: 127515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.366028] LustreError: 127515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4072.292973] LustreError: 130070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842475 with bad export cookie 13141709258223920240 [ 4072.299636] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4072.310732] LustreError: 130070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4072.825912] Lustre: server umount lustre-MDT0001 complete [ 4086.405458] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4090.393990] Key type lgssc unregistered [ 4090.906840] LNet: 131110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4090.928557] LNetError: 131110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4090.952605] LNet: Removed LNI 192.168.204.115@tcp [ 4092.071652] Key type .llcrypt unregistered [ 4092.075765] Key type ._llcrypt unregistered [ 4121.432980] Key type ._llcrypt registered [ 4121.446984] Key type .llcrypt registered [ 4121.598731] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4137.840931] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4138.756438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4138.853216] alg: No test for adler32 (adler32-zlib) [ 4139.972970] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 4140.476712] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4142.226573] Key type lgssc registered [ 4143.166879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4183.022531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4183.045991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.413574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4184.472677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4184.585569] Lustre: lustre-MDT0000: new disk, initializing [ 4184.719909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4184.744462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4189.655385] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4201.758464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.964516] Lustre: 134504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4202.017956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4202.023227] Lustre: Skipped 1 previous similar message [ 4202.083860] Lustre: lustre-MDT0001: new disk, initializing [ 4202.173758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4202.222736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4202.265955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4207.416044] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4218.505903] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4225.208573] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4232.833089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4233.237427] Lustre: lustre-OST0000: new disk, initializing [ 4233.244260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4233.249784] Lustre: 136474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4233.365110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.140715] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4243.593199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4243.614939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4243.730275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4250.121486] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4254.830889] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4255.044645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4258.880351] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4259.146602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4264.418478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4264.422660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.429134] LustreError: Skipped 1 previous similar message [ 4264.443516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4264.448312] Lustre: Skipped 1 previous similar message [ 4269.538313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.547175] Lustre: Skipped 1 previous similar message [ 4274.677625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.684594] Lustre: Skipped 1 previous similar message [ 4275.168140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4275.448040] Lustre: server umount lustre-OST0000 complete [ 4279.265162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4279.269074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.273235] Lustre: Skipped 1 previous similar message [ 4279.275248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4284.883943] Lustre: server umount lustre-MDT0000 complete [ 4288.434194] LustreError: 134497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842691 with bad export cookie 4893450516893076166 [ 4288.438474] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4288.443978] LustreError: 134497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4288.748436] Lustre: server umount lustre-MDT0001 complete [ 4298.488396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.958599] LustreError: 138693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4298.980299] LustreError: 138693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4299.025584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.048123] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4304.356745] LustreError: 138694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.374955] LustreError: 138694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4308.452272] LustreError: 138693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.569935] LustreError: 138694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.098267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.497238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4322.285761] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4332.331768] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4339.517951] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4347.158725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.527770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.474425] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4362.308669] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4367.702967] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4370.150827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4375.502604] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4375.665642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4385.853077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4386.135575] Lustre: lustre-OST0001: new disk, initializing [ 4386.141713] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4386.149989] Lustre: 142080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4386.307526] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4395.122618] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4396.713392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4396.727311] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4396.850853] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4404.740464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4404.928575] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4409.872484] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4410.117352] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4416.508333] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9da1c7c40800.ost_server_uuid 50 [ 4418.102248] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9da1c7c40800.ost_server_uuid in IDLE state after 0 sec [ 4427.755295] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.759831] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4427.776288] Lustre: Skipped 1 previous similar message [ 4427.796808] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4427.801389] Lustre: Skipped 2 previous similar messages [ 4432.870360] LustreError: 140517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.907574] Lustre: server umount lustre-OST0001 complete [ 4437.988330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4437.997064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.998933] LustreError: Skipped 1 previous similar message [ 4438.016286] Lustre: Skipped 1 previous similar message [ 4438.025344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4438.032818] Lustre: Skipped 1 previous similar message [ 4443.114402] LustreError: 140516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.144741] LustreError: 140516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4451.296281] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4451.487298] Lustre: server umount lustre-OST0000 complete [ 4458.466964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.479083] Lustre: Skipped 1 previous similar message [ 4458.485581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.495210] Lustre: Skipped 5 previous similar messages [ 4461.484390] Lustre: server umount lustre-MDT0000 complete [ 4463.586164] LustreError: 138693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.605513] LustreError: 138693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4465.299189] LustreError: 138675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842868 with bad export cookie 4893450516893077048 [ 4465.301608] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4465.307601] LustreError: 138675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4465.764470] Lustre: server umount lustre-MDT0001 complete [ 4489.882568] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4493.559819] Key type lgssc unregistered [ 4494.194658] LNet: 145085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4494.204116] LNetError: 145085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4494.219951] LNet: Removed LNI 192.168.204.115@tcp [ 4495.982824] Key type .llcrypt unregistered [ 4495.984640] Key type ._llcrypt unregistered [ 4545.309335] Key type ._llcrypt registered [ 4545.312840] Key type .llcrypt registered [ 4545.432515] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4559.524546] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4560.751130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4560.788179] alg: No test for adler32 (adler32-zlib) [ 4561.875906] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 4562.102757] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4563.744276] Key type lgssc registered [ 4564.858639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4604.806776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4604.836026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.157457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4606.203263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4606.321607] Lustre: lustre-MDT0000: new disk, initializing [ 4606.413578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4606.443491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4611.019862] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4622.945895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.081940] Lustre: 149240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4623.154938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4623.160412] Lustre: Skipped 1 previous similar message [ 4623.257836] Lustre: lustre-MDT0001: new disk, initializing [ 4623.384047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4623.419217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4623.437071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4627.794516] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4639.197369] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4646.837421] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4654.249789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.472500] Lustre: lustre-OST0000: new disk, initializing [ 4654.475490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4654.483366] Lustre: 151209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4654.592571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4660.379634] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4660.799691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4660.808449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4660.926224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4669.321538] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4676.121141] Lustre: Failing over lustre-OST0000 [ 4676.301270] Lustre: server umount lustre-OST0000 complete [ 4692.963525] Lustre: 147463:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843079/real 1788843079] req@00000000293668ff x1875737800902528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788843095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4692.964747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.989230] Lustre: 147463:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4693.017066] Lustre: Skipped 1 previous similar message [ 4696.953500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.288636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.321476] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4698.080192] Lustre: 147465:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843084/real 1788843084] req@00000000f549880c x1875737800903808/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788843100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4698.116253] Lustre: 147465:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4702.178088] Lustre: 147463:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843089/real 1788843089] req@000000000c0833e3 x1875737800904960/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788843105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4702.374131] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4702.843662] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4703.326327] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4703.326935] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4707.299834] Lustre: 147465:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843094/real 1788843094] req@000000007630aa25 x1875737800905856/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788843110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4707.334551] Lustre: 147465:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4707.513167] Lustre: Failing over lustre-MDT0000 [ 4707.756688] Lustre: server umount lustre-MDT0000 complete [ 4707.811720] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4712.929766] LustreError: 149251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.953063] LustreError: 149251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4713.525706] LustreError: 149248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4718.058363] LustreError: 150404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4718.088570] LustreError: 150404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4723.170489] LustreError: 149252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4723.198062] LustreError: 149252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4727.227959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.400348] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4727.649936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4727.716297] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4728.901500] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4732.652569] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4732.913729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4732.919119] Lustre: Skipped 1 previous similar message [ 4732.936051] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4732.992902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4743.148206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.151905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4743.159953] Lustre: Skipped 3 previous similar messages [ 4743.169137] Lustre: Skipped 1 previous similar message [ 4748.046227] Lustre: server umount lustre-OST0000 complete [ 4753.378203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.379302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4753.400698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4758.498770] LustreError: 149251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4758.531651] LustreError: 149251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4758.709336] Lustre: server umount lustre-MDT0000 complete [ 4763.406975] LustreError: 151208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843166 with bad export cookie 17110075347059349195 [ 4763.410113] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4763.423161] LustreError: 151208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4763.931472] Lustre: server umount lustre-MDT0001 complete [ 4776.677698] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4779.939421] Key type lgssc unregistered [ 4780.270188] LNet: 154864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4780.277485] LNetError: 154864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4780.294185] LNet: Removed LNI 192.168.204.115@tcp [ 4781.183638] Key type .llcrypt unregistered [ 4781.190317] Key type ._llcrypt unregistered [ 4822.164079] Key type ._llcrypt registered [ 4822.165834] Key type .llcrypt registered [ 4822.233657] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4823.202776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4823.256471] alg: No test for adler32 (adler32-zlib) [ 4824.366450] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 4824.639815] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4826.329437] Key type lgssc registered [ 4827.522665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4837.147454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4837.182858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4838.733444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4843.808731] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4848.098957] LustreError: 156385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.125483] LustreError: 156385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4853.219718] LustreError: 156386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.143910] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4858.343526] LustreError: 156385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.763202] LustreError: 156414:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4858.773764] LustreError: 156414:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4858.778505] LustreError: 156414:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4858.984851] Lustre: server umount lustre-MDT0000 complete [ 4875.121337] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4888.500376] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4930.737070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.101814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4931.157739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4931.263686] Lustre: lustre-MDT0000: new disk, initializing [ 4931.367835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4931.393876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4935.706585] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4946.516540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4946.689516] Lustre: 159722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4946.782175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4946.785520] Lustre: Skipped 1 previous similar message [ 4946.849908] Lustre: lustre-MDT0001: new disk, initializing [ 4946.957162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4946.988321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4947.003716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4952.286786] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4961.962933] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4968.327844] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4975.628723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4975.939290] Lustre: lustre-OST0000: new disk, initializing [ 4975.942682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4975.950196] Lustre: 161691:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4976.075345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4983.645955] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4984.417890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4984.425228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4984.537667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4993.906814] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4997.797405] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4999.886319] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:56:40 (1788843400) [ 5281.760925] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5281.771602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5281.811294] Lustre: Skipped 1 previous similar message [ 5281.817871] Lustre: Skipped 2 previous similar messages [ 5286.751165] Lustre: server umount lustre-MDT0000 complete [ 5286.881197] LustreError: 159732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.909516] LustreError: 159732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5291.315700] LustreError: 160519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843694 with bad export cookie 7651954244344289370 [ 5291.319366] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.332152] LustreError: 160519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5292.001869] LustreError: 159728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5292.003524] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.060016] Lustre: Skipped 1 previous similar message [ 5292.067949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5297.139735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5297.744369] Lustre: server umount lustre-MDT0001 complete [ 5308.586348] Lustre: server umount lustre-OST0000 complete [ 5319.515091] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5330.900729] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5371.845415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.232961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5372.276238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5372.405726] Lustre: lustre-MDT0000: new disk, initializing [ 5372.539284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5372.569768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5377.553858] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5388.705548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.883958] Lustre: 166374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5388.977992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5388.985448] Lustre: Skipped 1 previous similar message [ 5389.134705] Lustre: lustre-MDT0001: new disk, initializing [ 5389.268656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5389.309309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5389.326898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5394.627742] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5405.041847] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5411.844691] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5419.434135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5419.717563] Lustre: lustre-OST0000: new disk, initializing [ 5419.722782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5419.734852] Lustre: 168343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5419.808314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5425.735704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5425.742117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5425.847739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5426.683643] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5437.292905] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5443.161725] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5443.514378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5448.942708] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5449.192246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5451.745402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.747407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5451.774399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.790977] Lustre: Skipped 1 previous similar message [ 5456.871271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5456.889846] Lustre: Skipped 1 previous similar message [ 5457.536479] Lustre: server umount lustre-OST0000 complete [ 5461.998462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5462.037922] Lustre: Skipped 2 previous similar messages [ 5467.107758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5467.128041] Lustre: Skipped 3 previous similar messages [ 5476.321148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5476.553709] Lustre: server umount lustre-MDT0000 complete [ 5477.353725] LustreError: 166385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5477.376864] LustreError: 166385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5481.195034] LustreError: 166364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843884 with bad export cookie 7651954244344448200 [ 5481.197423] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.229249] LustreError: 166364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5481.596123] Lustre: server umount lustre-MDT0001 complete [ 5491.867243] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:04:52 (1788843892) [ 5514.065932] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5525.541350] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5568.884319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.152233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5569.216892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5569.307846] Lustre: lustre-MDT0000: new disk, initializing [ 5569.388394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5569.413837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5575.689848] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5588.298231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5588.413664] Lustre: 173101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5588.645202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5594.220331] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5605.080748] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5612.317085] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5620.159287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5620.624943] Lustre: lustre-OST0000: new disk, initializing [ 5620.632728] Lustre: Skipped 1 previous similar message [ 5620.640800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5620.645806] Lustre: Skipped 2 previous similar messages [ 5620.649883] Lustre: 175084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5628.369685] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5630.025323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5630.033813] Lustre: Skipped 1 previous similar message [ 5630.044233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5630.199865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5638.533944] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5650.914622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.919156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5650.936612] Lustre: Skipped 1 previous similar message [ 5650.952651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5650.954116] Lustre: Skipped 3 previous similar messages [ 5652.225443] Lustre: server umount lustre-OST0000 complete [ 5660.642368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5660.659982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.815887] Lustre: server umount lustre-MDT0000 complete [ 5666.274451] LustreError: 174480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5666.295703] LustreError: 174480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5667.367306] LustreError: 173094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844070 with bad export cookie 7651954244344449082 [ 5667.373540] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5667.391188] LustreError: 173094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5668.054945] Lustre: server umount lustre-MDT0001 complete [ 5678.750748] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:07:59 (1788844079) [ 5701.107127] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5713.284235] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5755.825140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.244253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5756.284870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5756.366614] Lustre: lustre-MDT0000: new disk, initializing [ 5756.444885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.453241] Lustre: Skipped 2 previous similar messages [ 5756.489764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5762.544618] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5775.395505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.557311] Lustre: 179410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5775.991411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5781.774502] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5792.951606] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5800.947667] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5809.246561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.722607] Lustre: 181390:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5816.923856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5817.045991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5817.262098] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5826.537078] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5837.799946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5837.806778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5837.818897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5837.818906] Lustre: Skipped 3 previous similar messages [ 5837.826959] Lustre: Skipped 2 previous similar messages [ 5837.857691] LustreError: Skipped 1 previous similar message [ 5839.432773] Lustre: server umount lustre-OST0000 complete [ 5847.521635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5858.272377] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5858.512946] Lustre: server umount lustre-MDT0000 complete [ 5862.371559] LustreError: 180486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.400876] LustreError: 180486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5863.225703] LustreError: 179403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844266 with bad export cookie 7651954244344450167 [ 5863.235387] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5863.238624] LustreError: 179403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5863.951121] Lustre: server umount lustre-MDT0001 complete [ 5878.880860] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5882.965816] Key type lgssc unregistered [ 5883.467965] LNet: 183246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5883.487750] LNetError: 183246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5883.518836] LNet: Removed LNI 192.168.204.115@tcp [ 5884.731928] Key type .llcrypt unregistered [ 5884.734246] Key type ._llcrypt unregistered [ 5914.512181] Key type ._llcrypt registered [ 5914.515312] Key type .llcrypt registered [ 5914.670656] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5915.557636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5915.579952] alg: No test for adler32 (adler32-zlib) [ 5916.584528] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 5916.822377] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5918.528162] Key type lgssc registered [ 5919.934160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5929.171899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5929.196812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.699761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.163953] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5940.194655] LustreError: 184659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.217828] LustreError: 184659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5945.316539] LustreError: 184660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.624750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5947.086517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5952.780577] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5961.791682] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5968.872900] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5976.627884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5977.082913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5984.920707] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5994.011809] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6000.626468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6021.093836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6021.102282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.103202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6021.124191] Lustre: Skipped 1 previous similar message [ 6026.027216] Lustre: server umount lustre-OST0000 complete [ 6031.332960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6031.335768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6031.345100] Lustre: Skipped 1 previous similar message [ 6031.355871] Lustre: Skipped 2 previous similar messages [ 6036.451687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6036.463923] Lustre: Skipped 1 previous similar message [ 6037.785566] Lustre: server umount lustre-MDT0000 complete [ 6041.570950] LustreError: 184656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.611483] LustreError: 184656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6042.553376] LustreError: 184641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844445 with bad export cookie 3879504012114609653 [ 6042.560255] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.565746] LustreError: 184641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6042.948709] Lustre: server umount lustre-MDT0001 complete [ 6057.276536] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6061.062546] Key type lgssc unregistered [ 6061.358454] LNet: 188217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6061.370512] LNetError: 188217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6061.401258] LNet: Removed LNI 192.168.204.115@tcp [ 6062.744764] Key type .llcrypt unregistered [ 6062.746326] Key type ._llcrypt unregistered [ 6092.995817] Key type ._llcrypt registered [ 6092.998660] Key type .llcrypt registered [ 6093.227755] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6095.276167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6095.345667] alg: No test for adler32 (adler32-zlib) [ 6096.630228] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 6097.061815] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6098.824174] Key type lgssc registered [ 6100.581151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6111.152996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6111.186676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6113.003818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6119.714495] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6122.491690] LustreError: 189632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.508885] LustreError: 189632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6127.588261] LustreError: 189631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6131.916804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.499733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6138.657730] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6147.529232] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6154.399858] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6162.456369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6162.813104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.898756] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6180.619699] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6182.273578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6191.716464] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 6192.617190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.622901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6192.625941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6192.643752] Lustre: Skipped 1 previous similar message [ 6192.681939] LustreError: Skipped 1 previous similar message [ 6196.811958] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 6196.821399] Lustre: Skipped 1 previous similar message [ 6197.523954] Lustre: server umount lustre-OST0000 complete [ 6206.929478] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9da1d92ab000.ost_server_uuid 50 [ 6209.184363] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6224.868622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6224.877094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.890076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6228.449415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6230.422656] Lustre: server umount lustre-MDT0000 complete [ 6233.570979] LustreError: 189631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.597373] LustreError: 189631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6234.890785] LustreError: 190366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844637 with bad export cookie 1914625589012718463 [ 6234.912199] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6235.400111] Lustre: server umount lustre-MDT0001 complete [ 6247.490962] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:17:27 (1788844647) [ 6256.784173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.442719] LustreError: 193552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.473148] LustreError: 193552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6257.607501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.756449] LustreError: 193553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.114446] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6272.994792] LustreError: 193552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6273.004881] LustreError: 193552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6273.373780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.801706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6278.614881] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6288.343904] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6295.600924] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6303.359536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6303.765995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6310.774319] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6319.817413] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6327.368598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6331.710233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6332.089703] Lustre: lustre-OST0001: new disk, initializing [ 6332.101485] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6332.110279] Lustre: 196395:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6332.283187] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6337.648947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6337.663975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6337.797458] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6338.217226] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6350.212486] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6353.890369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6353.897742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6353.916477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6353.923731] Lustre: Skipped 2 previous similar messages [ 6357.995133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6358.011116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.024873] Lustre: server umount lustre-OST0000 complete [ 6361.145529] LustreError: 196403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.440967] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6368.731927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6375.154864] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6375.373561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6394.341529] LustreError: 196403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6394.346267] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.346835] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.361751] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6394.361759] Lustre: Skipped 4 previous similar messages [ 6394.385197] LustreError: 196403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 6394.399251] Lustre: Skipped 1 previous similar message [ 6396.725713] Lustre: server umount lustre-OST0001 complete [ 6407.917889] Lustre: server umount lustre-MDT0000 complete [ 6411.410678] LustreError: 193532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844814 with bad export cookie 1914625589012719534 [ 6411.420792] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6411.434352] LustreError: 193532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6411.868234] Lustre: server umount lustre-MDT0001 complete [ 6445.940970] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:20:45 (1788844845) [ 6454.268136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.586175] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6454.614646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6454.773446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6459.359378] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6470.040557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.231653] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6470.327553] Lustre: 199835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6476.278960] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6484.683784] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6490.804448] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6498.402853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6498.646956] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6506.223436] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6507.210578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6515.041462] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6523.016478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.087753] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6523.355180] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6523.361102] Lustre: Skipped 2 previous similar messages [ 6528.715704] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6530.721804] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6540.330703] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6544.368388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.370411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6544.390665] Lustre: Skipped 3 previous similar messages [ 6544.406707] Lustre: Skipped 5 previous similar messages [ 6549.490413] LustreError: 201684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6549.521478] LustreError: 201684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6549.585503] Lustre: server umount lustre-OST0000 complete [ 6569.954591] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6569.972034] LustreError: Skipped 1 previous similar message [ 6572.619569] Lustre: server umount lustre-OST0001 complete [ 6578.145311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.157505] Lustre: Skipped 2 previous similar messages [ 6578.162762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6578.168631] Lustre: Skipped 3 previous similar messages [ 6584.038757] Lustre: server umount lustre-MDT0000 complete [ 6588.221140] LustreError: 199829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844991 with bad export cookie 1914625589012720941 [ 6588.230632] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6588.241181] LustreError: 199829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6588.651817] Lustre: server umount lustre-MDT0001 complete [ 6615.894661] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:23:36 (1788845016) [ 6634.074086] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6644.750448] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6688.656366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6688.905494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6688.927906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6689.012896] Lustre: lustre-MDT0000: new disk, initializing [ 6689.090201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.114629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6694.616144] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6707.218990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.329861] Lustre: 207936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6707.399812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6707.404378] Lustre: Skipped 1 previous similar message [ 6707.541958] Lustre: lustre-MDT0001: new disk, initializing [ 6707.714452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6707.734897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6713.582843] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6726.141457] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6733.299181] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6741.216027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6741.531210] Lustre: lustre-OST0000: new disk, initializing [ 6741.534921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6741.556892] Lustre: 209909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6749.455238] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6749.810411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6749.844169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6750.025476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6760.631093] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6766.467625] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6766.718672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6771.697701] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6771.938690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6775.783193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.800436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6775.806626] Lustre: Skipped 1 previous similar message [ 6775.808118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6775.808126] Lustre: Skipped 2 previous similar messages [ 6775.852788] LustreError: Skipped 2 previous similar messages [ 6780.331091] Lustre: server umount lustre-OST0000 complete [ 6799.328127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6799.606864] Lustre: server umount lustre-MDT0000 complete [ 6800.358985] LustreError: 207946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6800.395940] LustreError: 207946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6804.211716] LustreError: 207926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845207 with bad export cookie 1914625589012722719 [ 6804.215378] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6804.220210] LustreError: 207926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6804.677751] Lustre: server umount lustre-MDT0001 complete [ 6814.866039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.613524] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6830.563926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.319654] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6844.796778] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6851.222563] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6859.050212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6866.455269] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6874.933633] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6879.258111] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6879.438484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6883.947219] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6884.165173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6886.372495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6886.380178] LustreError: Skipped 1 previous similar message [ 6900.192224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6900.553258] Lustre: server umount lustre-OST0000 complete [ 6908.823983] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6914.795104] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6946.234247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6946.515958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6946.522103] Lustre: Skipped 5 previous similar messages [ 6951.692334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6951.711363] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6951.717390] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6951.756434] Lustre: Skipped 1 previous similar message [ 6954.088455] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6962.906842] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6964.882663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6972.385195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.388927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6972.399907] LustreError: Skipped 2 previous similar messages [ 6972.400655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.400659] Lustre: Skipped 13 previous similar messages [ 6972.425254] Lustre: Skipped 6 previous similar messages [ 6975.726235] Lustre: server umount lustre-OST0000 complete [ 6986.898358] Lustre: server umount lustre-MDT0000 complete [ 6991.214818] LustreError: 212107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845394 with bad export cookie 1914625589012723608 [ 6991.226731] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6991.235517] LustreError: 212107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6991.890674] Lustre: server umount lustre-MDT0001 complete [ 7001.449609] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:30:02 (1788845402) [ 7008.731722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7014.118548] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7023.096429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.979229] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7036.401518] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7043.608526] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7051.235530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7058.911704] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7068.342177] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7073.719621] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7074.076540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7079.156219] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7079.415373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7087.261284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7087.521870] Lustre: lustre-OST0001: new disk, initializing [ 7087.526629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7087.531207] Lustre: 221469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7093.865613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7093.874240] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7093.942945] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7094.798737] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7104.545995] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7108.642753] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7108.896571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7112.853841] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7113.050566] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7119.849698] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7119.866299] LustreError: Skipped 1 previous similar message [ 7121.212583] Lustre: server umount lustre-OST0001 complete [ 7124.967772] LustreError: 221727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.982728] LustreError: 221727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7127.416419] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7127.641683] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7132.016589] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7132.196401] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7161.569272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7166.957198] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7166.970787] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7166.972940] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7166.996035] Lustre: Skipped 1 previous similar message [ 7168.319618] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7176.604685] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7178.324794] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7186.318342] Lustre: server umount lustre-OST0001 complete [ 7210.301294] Lustre: server umount lustre-OST0000 complete [ 7220.544636] Lustre: server umount lustre-MDT0000 complete [ 7224.606234] LustreError: 221468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845627 with bad export cookie 1914625589012724574 [ 7224.611203] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7224.618336] LustreError: 221468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7231.184230] Lustre: server umount lustre-MDT0001 complete [ 7259.662783] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:34:20 (1788845660) [ 7266.294860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7266.603968] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7266.637658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7271.304834] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7279.788495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.913264] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7279.939179] Lustre: 226712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7285.602755] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7294.605458] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7301.316979] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7309.417718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7309.669660] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7316.771619] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7325.147506] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7335.929428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7336.043371] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7344.689093] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7353.240443] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7355.232192] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7360.509432] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7360.776113] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7365.196599] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7365.435545] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7371.955753] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9da1d800c800.ost_server_uuid 50 [ 7374.123254] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9da1d800c800.ost_server_uuid in IDLE state after 0 sec [ 7375.244776] Lustre: Permanently deactivating lustre-OST0001 [ 7375.257359] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7379.561869] Lustre: Permanently reactivating lustre-OST0001 [ 7379.578574] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7379.587517] Lustre: Skipped 2 previous similar messages [ 7385.066393] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7385.080982] LustreError: Skipped 4 previous similar messages [ 7385.094400] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.107537] Lustre: Skipped 10 previous similar messages [ 7385.112391] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7385.120913] Lustre: Skipped 11 previous similar messages [ 7390.650783] Lustre: server umount lustre-OST0001 complete [ 7401.085858] Lustre: server umount lustre-OST0000 complete [ 7419.360331] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7419.961668] Lustre: server umount lustre-MDT0000 complete [ 7424.621928] LustreError: 226705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845827 with bad export cookie 1914625589012725862 [ 7424.631671] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7424.642289] LustreError: 226705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7425.107387] Lustre: server umount lustre-MDT0001 complete [ 7455.178647] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:37:35 (1788845855) [ 7463.502886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.744557] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7463.789672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7463.800481] Lustre: Skipped 2 previous similar messages [ 7464.050340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7464.069037] Lustre: Skipped 9 previous similar messages [ 7468.316808] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7475.795332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.980920] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7476.036799] Lustre: 233408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7481.249843] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7490.759341] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7498.574640] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7506.234810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7506.505331] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7506.553100] Lustre: Permanently deactivating lustre-OST0000 [ 7506.578281] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7512.667763] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7514.138541] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7523.558834] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7531.801828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7531.921579] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7539.261103] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7542.623504] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7549.484730] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7552.262314] Lustre: Permanently reactivating lustre-OST0000 [ 7552.272670] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7552.276788] Lustre: Skipped 2 previous similar messages [ 7562.296917] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7562.299776] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 50 seconds. I think it's dead, and I am evicting it. exp 00000000cfdbf6f5, cur 1788845965 deadline 1788845945 last 1788845915 [ 7562.310898] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7562.338259] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7562.378825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7582.176385] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7582.347763] Lustre: server umount lustre-OST0001 complete [ 7592.721752] Lustre: server umount lustre-OST0000 complete [ 7611.360455] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7611.540283] Lustre: server umount lustre-MDT0000 complete [ 7615.345154] LustreError: 234150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846018 with bad export cookie 1914625589012727535 [ 7615.358305] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7615.364446] LustreError: 234150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7615.676592] Lustre: server umount lustre-MDT0001 complete [ 7625.602398] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7628.205282] Key type lgssc unregistered [ 7628.591639] LNet: 238293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7628.607323] LNetError: 238293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7628.645820] LNet: Removed LNI 192.168.204.115@tcp [ 7629.680359] Key type .llcrypt unregistered [ 7629.683248] Key type ._llcrypt unregistered [ 7657.083311] Key type ._llcrypt registered [ 7657.088276] Key type .llcrypt registered [ 7657.151663] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7658.201863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7658.284145] alg: No test for adler32 (adler32-zlib) [ 7659.373285] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 7659.630351] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7661.344186] Key type lgssc registered [ 7662.421890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7673.819322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7673.854850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.284151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7680.451179] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7685.606667] LustreError: 239769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7685.626337] LustreError: 239769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7690.246776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.387872] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7690.417684] Lustre: 239759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7690.426511] Lustre: Permanently deactivating lustre-MDT0001 [ 7690.439585] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7690.718047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7693.601790] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7693.609871] LustreError: 239800:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7693.623800] LustreError: 239800:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7693.636437] LustreError: 239800:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7695.637798] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7703.795741] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7709.938092] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7717.912443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7718.271563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7720.319939] LustreError: 241610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7720.367559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7725.319237] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7725.539913] LustreError: 241610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7730.665988] LustreError: 241714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7733.519630] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7735.781534] LustreError: 241612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7735.809492] LustreError: 241612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7741.322311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7741.485210] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7746.579938] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7746.590503] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7748.098623] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7756.404325] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7760.951636] Lustre: Permanently reactivating lustre-MDT0001 [ 7760.986231] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7760.993413] Lustre: Skipped 2 previous similar messages [ 7765.584407] Lustre: lustre-MDT0001: Client b2405b67-7b6e-4cfc-aeb8-a94d12594051 (at 192.168.204.15@tcp) reconnecting [ 7772.682118] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7774.530260] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7781.389500] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7781.752607] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7787.392392] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7787.655964] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7790.401963] Lustre: Permanently deactivating lustre-MDT0001 [ 7790.428349] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7790.436991] Lustre: Skipped 2 previous similar messages [ 7800.367127] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7813.607246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7813.617786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.636973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7815.139036] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.141662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7815.166670] Lustre: Skipped 1 previous similar message [ 7815.176821] Lustre: Skipped 1 previous similar message [ 7816.918734] Lustre: server umount lustre-MDT0000 complete [ 7820.265382] LustreError: 239765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.291728] LustreError: 239765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7821.555093] LustreError: 239751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846224 with bad export cookie 1488900825032629988 [ 7821.558671] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7821.567461] LustreError: 239751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7821.913142] Lustre: server umount lustre-MDT0001 complete [ 7828.065993] Lustre: server umount lustre-OST0000 complete [ 7840.487548] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:44:00 (1788846240) [ 7853.425861] Lustre: server umount lustre-OST0001 complete [ 7862.846103] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 7874.917965] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7918.600568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.768190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7918.776886] Lustre: Skipped 2 previous similar messages [ 7918.796864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7918.855370] Lustre: lustre-MDT0000: new disk, initializing [ 7918.929718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7918.957652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7924.777648] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7937.034433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7937.155626] Lustre: 248081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7937.207725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7937.211420] Lustre: Skipped 1 previous similar message [ 7937.276922] Lustre: lustre-MDT0001: new disk, initializing [ 7937.383095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7937.418736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7937.436686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7942.564793] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7953.789480] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7960.503966] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7969.054220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7969.485120] Lustre: lustre-OST0000: new disk, initializing [ 7969.495583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7969.501842] Lustre: 250052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7969.596701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7975.004570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7975.022929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7975.218429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7977.272220] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7989.502836] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7991.971288] LustreError: 248088:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8001.979308] LustreError: 248088:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8002.026630] LustreError: 248089:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8002.600990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.890782] Lustre: lustre-OST0001: new disk, initializing [ 8002.899557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8002.910836] Lustre: 251172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8003.058146] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8010.383077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8010.395278] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8010.449961] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8010.495317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8012.097156] LustreError: 248089:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8021.090958] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8023.076289] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8026.081885] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8026.086232] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.104927] LustreError: Skipped 1 previous similar message [ 8026.123639] Lustre: Skipped 1 previous similar message [ 8026.143589] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8026.150062] Lustre: Skipped 3 previous similar messages [ 8031.216981] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8031.228028] Lustre: Skipped 1 previous similar message [ 8031.470237] Lustre: server umount lustre-OST0001 complete [ 8036.329769] LustreError: 250037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8036.363811] LustreError: 250037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8051.689584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8051.692885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8051.704631] Lustre: Skipped 2 previous similar messages [ 8055.498953] Lustre: server umount lustre-OST0000 complete [ 8060.386279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8060.398500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8060.436110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8060.446924] Lustre: Skipped 1 previous similar message [ 8065.635000] Lustre: server umount lustre-MDT0000 complete [ 8069.543605] LustreError: 248074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846472 with bad export cookie 1488900825032632578 [ 8069.548948] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8069.551381] LustreError: 248074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8075.949369] Lustre: server umount lustre-MDT0001 complete [ 8087.189745] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8090.110161] Key type lgssc unregistered [ 8090.356494] LNet: 253204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8090.366486] LNetError: 253204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8090.393594] LNet: Removed LNI 192.168.204.115@tcp [ 8091.411824] Key type .llcrypt unregistered [ 8091.417862] Key type ._llcrypt unregistered [ 8140.078760] Key type ._llcrypt registered [ 8140.082562] Key type .llcrypt registered [ 8140.232450] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8141.009147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8141.054960] alg: No test for adler32 (adler32-zlib) [ 8142.196763] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 8142.641688] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8144.384413] Key type lgssc registered [ 8145.652916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8154.922992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8154.942603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8156.177479] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8156.242966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8156.641903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8161.492710] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8170.509731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.639173] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8170.660724] Lustre: 255410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8170.933051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8177.033832] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8184.657743] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8192.624792] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8202.529341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8202.715712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8202.990247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8208.172112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8211.774896] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8221.902851] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8279.523404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8279.535265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8279.578243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8280.033882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8280.052578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8280.080716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8280.084705] Lustre: Skipped 1 previous similar message [ 8285.159939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8285.171636] Lustre: Skipped 1 previous similar message [ 8285.509379] Lustre: server umount lustre-OST0000 complete [ 8291.055852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8305.269943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8305.661304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8310.764794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8310.780567] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8312.444794] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8315.941628] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8315.967514] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8321.569828] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8367.076773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8367.079711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.096382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8367.102849] LustreError: Skipped 1 previous similar message [ 8367.135496] Lustre: Skipped 1 previous similar message [ 8369.365266] Lustre: server umount lustre-OST0000 complete [ 8375.780940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8375.793889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.811644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.817243] Lustre: Skipped 1 previous similar message [ 8380.082944] Lustre: server umount lustre-MDT0000 complete [ 8384.240246] LustreError: 255402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846787 with bad export cookie 5347874302262788404 [ 8384.250489] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8384.268543] LustreError: 255402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8384.654737] Lustre: server umount lustre-MDT0001 complete [ 8398.216937] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8400.942885] Key type lgssc unregistered [ 8401.288682] LNet: 260835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8401.301475] LNetError: 260835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8401.331119] LNet: Removed LNI 192.168.204.115@tcp [ 8402.203644] Key type .llcrypt unregistered [ 8402.206425] Key type ._llcrypt unregistered [ 8428.844881] Key type ._llcrypt registered [ 8428.848305] Key type .llcrypt registered [ 8428.945319] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8430.194782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8430.257325] alg: No test for adler32 (adler32-zlib) [ 8431.402161] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 8431.729738] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8433.482252] Key type lgssc registered [ 8434.612681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8442.832657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8442.870516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.395988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8448.941176] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8454.628051] LustreError: 262249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8454.650273] LustreError: 262249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8458.868143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8459.201410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8464.332132] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8473.605791] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8480.606431] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8489.352796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8489.794226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8490.796899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8496.595647] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8506.191767] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8531.944113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8531.969184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.992895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8535.015251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8535.016891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8537.208167] Lustre: server umount lustre-OST0000 complete [ 8545.251063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8545.273799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8545.278887] Lustre: Skipped 2 previous similar messages [ 8548.132150] Lustre: server umount lustre-MDT0000 complete [ 8550.370411] LustreError: 262246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8550.400198] LustreError: 262246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8552.925682] LustreError: 262231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846955 with bad export cookie 13838725151607952600 [ 8552.939843] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8552.948031] LustreError: 262231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8553.622463] Lustre: server umount lustre-MDT0001 complete [ 8569.442171] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8572.037613] Key type lgssc unregistered [ 8572.281367] LNet: 266554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8572.295813] LNetError: 266554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8572.307696] LNet: Removed LNI 192.168.204.115@tcp [ 8573.114981] Key type .llcrypt unregistered [ 8573.118966] Key type ._llcrypt unregistered [ 8596.464827] Key type ._llcrypt registered [ 8596.467785] Key type .llcrypt registered [ 8596.634887] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8598.036165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8598.101114] alg: No test for adler32 (adler32-zlib) [ 8599.297926] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 8599.608830] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8601.305140] Key type lgssc registered [ 8602.694191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8612.188783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8612.232414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8613.665380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8618.077477] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8624.100255] LustreError: 267780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8624.111943] LustreError: 267780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8627.610651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8628.012419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8632.673297] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8640.880324] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8647.513365] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8655.027742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.431407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.507812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8662.756735] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8672.141311] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8688.115121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8688.120715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.145397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.226420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.242790] Lustre: Skipped 1 previous similar message [ 8698.355909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8698.366745] Lustre: Skipped 2 previous similar messages [ 8699.874660] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8700.119907] Lustre: server umount lustre-OST0000 complete [ 8704.996494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8705.014530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8705.037510] Lustre: Skipped 1 previous similar message [ 8705.051699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8708.590196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8713.706951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8713.715022] Lustre: Skipped 3 previous similar messages [ 8719.330049] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8719.583908] Lustre: server umount lustre-MDT0000 complete [ 8723.940646] LustreError: 267781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8723.970165] LustreError: 267781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8724.017289] LustreError: 267761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847126 with bad export cookie 3365828465516447965 [ 8724.020314] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8724.031741] LustreError: 267761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8724.413957] Lustre: server umount lustre-MDT0001 complete [ 8737.045256] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8740.807797] Key type lgssc unregistered [ 8741.162981] LNet: 271465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8741.175482] LNetError: 271465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8741.192081] LNet: Removed LNI 192.168.204.115@tcp [ 8742.281280] Key type .llcrypt unregistered [ 8742.287523] Key type ._llcrypt unregistered [ 8773.308738] Key type ._llcrypt registered [ 8773.314121] Key type .llcrypt registered [ 8773.478324] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8775.280696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8775.315792] alg: No test for adler32 (adler32-zlib) [ 8776.393741] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 8776.670457] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8778.353574] Key type lgssc registered [ 8780.117740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8788.802722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8788.820400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8790.206699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8795.240146] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8800.740273] LustreError: 272878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8800.756091] LustreError: 272878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8805.527681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8805.859976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8806.063457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8812.208108] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8821.801920] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8829.242835] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8839.010581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8839.578813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8847.104921] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8858.174620] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8861.052952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8886.755526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.760108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8886.772422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.781485] Lustre: Skipped 1 previous similar message [ 8891.885124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8891.893126] Lustre: Skipped 1 previous similar message [ 8892.941122] Lustre: server umount lustre-OST0000 complete [ 8896.995868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8897.005571] Lustre: Skipped 1 previous similar message [ 8897.012264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8902.113241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8902.120335] Lustre: Skipped 2 previous similar messages [ 8911.328145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8911.579921] Lustre: server umount lustre-MDT0000 complete [ 8912.354639] LustreError: 273975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8912.382959] LustreError: 273975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8915.808379] LustreError: 272860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847318 with bad export cookie 3423111993234046383 [ 8915.830503] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8915.832426] LustreError: 272860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8916.268533] Lustre: server umount lustre-MDT0001 complete [ 8930.514688] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8933.521246] Key type lgssc unregistered [ 8933.895420] LNet: 277199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8933.899314] LNetError: 277199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8933.923941] LNet: Removed LNI 192.168.204.115@tcp [ 8934.972883] Key type .llcrypt unregistered [ 8934.974375] Key type ._llcrypt unregistered [ 8953.412260] Key type ._llcrypt registered [ 8953.414373] Key type .llcrypt registered [ 8953.544077] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8954.684248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8954.718924] alg: No test for adler32 (adler32-zlib) [ 8955.826818] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 8956.094195] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8957.792669] Key type lgssc registered [ 8959.388420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8967.584668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8967.608297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8969.015508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8974.375859] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8978.404391] LustreError: 278434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.429169] LustreError: 278434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8983.530724] LustreError: 278436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8985.115506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.442766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8990.045841] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8998.719384] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9004.962205] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9012.645536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9013.067594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9014.072190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9021.124320] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9029.964994] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9044.969184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9044.977287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.994625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9045.985674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9045.991572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9046.004809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9046.057801] Lustre: Skipped 1 previous similar message [ 9049.780727] Lustre: server umount lustre-OST0000 complete [ 9056.242777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9056.246293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9056.253747] Lustre: Skipped 1 previous similar message [ 9056.263770] Lustre: Skipped 1 previous similar message [ 9061.347544] LustreError: 278419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.369083] LustreError: 278419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9062.296550] Lustre: server umount lustre-MDT0000 complete [ 9066.467284] LustreError: 278441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9066.485414] LustreError: 278441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9066.630202] LustreError: 278399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847469 with bad export cookie 15988701978066634214 [ 9066.637969] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9067.148163] Lustre: server umount lustre-MDT0001 complete [ 9078.794181] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9082.098515] Key type lgssc unregistered [ 9082.375800] LNet: 282136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9082.383546] LNetError: 282136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9082.409696] LNet: Removed LNI 192.168.204.115@tcp [ 9083.610271] Key type .llcrypt unregistered [ 9083.617159] Key type ._llcrypt unregistered [ 9096.394312] Key type ._llcrypt registered [ 9096.396116] Key type .llcrypt registered [ 9096.558735] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9114.310653] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 9132.157887] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9133.731613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9133.749741] alg: No test for adler32 (adler32-zlib) [ 9134.744438] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 9134.984803] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9136.665840] Key type lgssc registered [ 9137.831736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9180.485861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9180.512202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9181.701324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9181.734811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9181.803834] Lustre: lustre-MDT0000: new disk, initializing [ 9181.939540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9181.959703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9186.744635] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9197.653348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.826509] Lustre: 285690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9197.899320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9197.902160] Lustre: Skipped 1 previous similar message [ 9197.966937] Lustre: lustre-MDT0001: new disk, initializing [ 9198.066933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9198.119861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9198.127082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9203.810824] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9216.535877] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9222.535946] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9230.718605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9231.013343] Lustre: lustre-OST0000: new disk, initializing [ 9231.021259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9231.026614] Lustre: 287658:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9231.129899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9237.583633] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9240.234956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9240.250923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9240.403764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9248.675718] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9253.735908] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9254.029636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9260.113426] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9260.546528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9266.149258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9266.155898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9266.157862] LustreError: Skipped 1 previous similar message [ 9266.161701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9266.180385] Lustre: Skipped 1 previous similar message [ 9266.199890] Lustre: Skipped 1 previous similar message [ 9268.921603] Lustre: server umount lustre-OST0000 complete [ 9274.851329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9274.859291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9274.885076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9276.385275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9276.387789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9279.436815] Lustre: server umount lustre-MDT0000 complete [ 9283.357994] LustreError: 285683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847686 with bad export cookie 13035945629978009508 [ 9283.363464] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9283.376514] LustreError: 285683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9283.747933] Lustre: server umount lustre-MDT0001 complete [ 9293.368934] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:08:13 (1788847693) [ 9301.111761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9301.467891] LustreError: 290073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9301.487884] LustreError: 290073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9301.546265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9306.222581] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9306.593893] LustreError: 290074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9311.715736] LustreError: 290073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9315.657171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.097855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9320.877600] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9329.492773] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9336.319860] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9344.072687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9344.533032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9351.635279] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9359.290749] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9371.118106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9371.120375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9371.132222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9371.149687] Lustre: Skipped 2 previous similar messages [ 9374.725497] Lustre: server umount lustre-OST0000 complete [ 9381.356049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.365448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9381.378961] Lustre: Skipped 2 previous similar messages [ 9381.397572] Lustre: Skipped 1 previous similar message [ 9384.852823] Lustre: server umount lustre-MDT0000 complete [ 9386.477181] LustreError: 290070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9386.519560] LustreError: 290070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9389.181108] LustreError: 291917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847792 with bad export cookie 13035945629978010390 [ 9389.196565] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9389.198081] LustreError: 291917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9389.555254] Lustre: server umount lustre-MDT0001 complete [ 9401.995455] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9405.296790] Key type lgssc unregistered [ 9405.574720] LNet: 293608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9405.582122] LNetError: 293608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9405.614859] LNet: Removed LNI 192.168.204.115@tcp [ 9406.507840] Key type .llcrypt unregistered [ 9406.510905] Key type ._llcrypt unregistered [ 9436.512664] Key type ._llcrypt registered [ 9436.516638] Key type .llcrypt registered [ 9436.643310] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 9452.735809] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9453.864526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9453.936557] alg: No test for adler32 (adler32-zlib) [ 9455.100580] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 9455.348413] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9456.987450] Key type lgssc registered [ 9458.149876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9500.365083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9500.394773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.706807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9501.751635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9501.829453] Lustre: lustre-MDT0000: new disk, initializing [ 9501.902314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9501.929146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9506.745603] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9518.710110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.891285] Lustre: 297438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9518.965581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9518.973829] Lustre: Skipped 1 previous similar message [ 9519.113784] Lustre: lustre-MDT0001: new disk, initializing [ 9519.245923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9519.293248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9519.309778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9524.243267] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9534.476119] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9542.361245] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9551.652271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9551.912742] Lustre: lustre-OST2710: new disk, initializing [ 9551.915740] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9551.921854] Lustre: 299408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9552.019115] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9558.182763] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9559.080079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9559.092657] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9559.169082] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9567.679193] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9575.368299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9575.518508] Lustre: lustre-OST03e8: new disk, initializing [ 9575.522385] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9575.528821] Lustre: 300472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9575.652462] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9582.179472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9582.184726] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9582.326214] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9582.510118] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9591.530977] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9600.124582] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9600.366986] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9605.515784] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9605.791602] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9610.429903] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9610.686217] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9615.307850] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9615.516426] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9621.475038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9621.487996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9621.511641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9621.741137] Lustre: server umount lustre-MDT0000 complete [ 9623.521969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9623.535289] Lustre: Skipped 2 previous similar messages [ 9625.359542] LustreError: 297428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848028 with bad export cookie 10682013355581398 [ 9625.361172] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9625.368321] LustreError: 297428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9625.635727] Lustre: server umount lustre-MDT0001 complete [ 9640.208454] Lustre: server umount lustre-OST2710 complete [ 9655.018922] Lustre: server umount lustre-OST03e8 complete [ 9676.543220] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 9687.902728] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9730.985189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.184523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9731.206492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9731.305508] Lustre: lustre-MDT0000: new disk, initializing [ 9731.443178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9731.462248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9736.372809] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9747.146195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.275869] Lustre: 305722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9747.379670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9747.384923] Lustre: Skipped 1 previous similar message [ 9747.502847] Lustre: lustre-MDT0001: new disk, initializing [ 9747.590295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9747.629952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9747.649734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9752.240977] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9762.044859] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9768.908516] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9776.462166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.825979] Lustre: lustre-OST0000: new disk, initializing [ 9776.835913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9776.857745] Lustre: 307694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9777.019034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9783.398941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9783.421960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9783.536460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9784.738815] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9798.404166] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9804.592981] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9804.870186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9809.953534] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9810.159938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9814.506383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9814.507938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9814.531668] LustreError: Skipped 1 previous similar message [ 9814.532428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9814.578834] Lustre: Skipped 1 previous similar message [ 9818.575729] Lustre: server umount lustre-OST0000 complete [ 9824.737440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9824.742122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.745548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9824.760032] Lustre: Skipped 1 previous similar message [ 9824.770554] Lustre: Skipped 2 previous similar messages [ 9829.610985] Lustre: server umount lustre-MDT0000 complete [ 9829.871508] LustreError: 305730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9829.909726] LustreError: 305730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9834.090719] LustreError: 305715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848237 with bad export cookie 10682013355583078 [ 9834.096810] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9834.105516] LustreError: 305715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9834.593665] Lustre: server umount lustre-MDT0001 complete [ 9844.651738] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:17:25 (1788848245) [ 9846.825359] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9849.566983] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:17:29 (1788848249) [ 9854.242450] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9882.638799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9882.989629] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9883.021729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9883.231995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9887.541561] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9896.297547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9896.427887] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9896.471328] Lustre: 311555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9901.283332] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9924.015601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9924.351075] Lustre: Denying initial registration attempt from nid 192.168.204.115@tcp, specified as failover [ 9924.374381] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9924.388668] LustreError: 313011:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9924.399592] LustreError: 313011:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9924.406479] LustreError: 313011:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9924.484357] Lustre: server umount lustre-OST0000 complete [ 9924.486541] LustreError: 313011:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9932.773246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9932.783800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9932.796238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9937.833284] Lustre: server umount lustre-MDT0000 complete [ 9938.965353] LustreError: 312380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9938.984868] LustreError: 312380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9942.039085] LustreError: 311548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848344 with bad export cookie 10682013355583960 [ 9942.041233] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9942.051268] LustreError: 311548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9942.354711] Lustre: server umount lustre-MDT0001 complete [ 9955.359826] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9958.568776] Key type lgssc unregistered [ 9958.910916] LNet: 314131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9958.922683] LNetError: 314131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9958.972045] LNet: Removed LNI 192.168.204.115@tcp [ 9960.037142] Key type .llcrypt unregistered [ 9960.039830] Key type ._llcrypt unregistered [ 9973.461219] Key type ._llcrypt registered [ 9973.466528] Key type .llcrypt registered [ 9973.615884] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9975.205756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9975.307174] alg: No test for adler32 (adler32-zlib) [ 9976.560409] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [ 9976.936841] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9978.664224] Key type lgssc registered [ 9979.980825] Lustre: Echo OBD driver; http://www.lustre.org/ [10033.792743] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [10045.581496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10045.623032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.911177] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10046.935926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10047.166063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10053.056548] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10064.235179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.338969] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10064.406091] Lustre: 316895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10064.708825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10070.556845] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10080.451084] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10087.415394] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10108.863697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10109.258676] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10109.592293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10117.957213] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10126.886661] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10131.938254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10131.942296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10131.948603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10131.952494] Lustre: Skipped 1 previous similar message [10135.333145] Lustre: server umount lustre-OST0000 complete [10141.667151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10141.671304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10141.693268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10141.695578] Lustre: Skipped 1 previous similar message [10147.297513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10147.321968] Lustre: Skipped 3 previous similar messages [10147.859333] Lustre: server umount lustre-MDT0000 complete [10154.402220] LustreError: 316887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848557 with bad export cookie 5582207948468428724 [10154.404894] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10154.407811] LustreError: 316887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10154.746242] Lustre: server umount lustre-MDT0001 complete [10168.843648] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [10171.801438] Key type lgssc unregistered [10172.138732] LNet: 320471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10172.144793] LNetError: 320471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10172.167052] LNet: Removed LNI 192.168.204.115@tcp [10173.134471] Key type .llcrypt unregistered [10173.136646] Key type ._llcrypt unregistered [10198.868267] Key type ._llcrypt registered [10198.871730] Key type .llcrypt registered [10198.997612] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [10200.535429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10200.650731] alg: No test for adler32 (adler32-zlib) [10201.858610] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [10202.213870] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [10203.952730] Key type lgssc registered [10205.212137] Lustre: Echo OBD driver; http://www.lustre.org/ [10213.297699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10213.309369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.779764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10219.561865] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10224.100532] LustreError: 321883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10224.128125] LustreError: 321883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10228.059404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.433104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10233.610748] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10240.279995] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10246.345602] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10256.690740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10257.166309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10264.077704] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10272.533949] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10284.005430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10284.009949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10284.036840] Lustre: Skipped 2 previous similar messages [10284.069138] Lustre: Skipped 2 previous similar messages [10285.672527] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [10289.126022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10289.130884] Lustre: Skipped 2 previous similar messages [10289.344439] Lustre: server umount lustre-MDT0000 complete [10290.748934] LustreError: 321878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10293.166633] LustreError: 321864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848696 with bad export cookie 18361756856440065400 [10293.177421] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10294.242623] LustreError: 323316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10294.242868] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10294.283890] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10298.274406] LustreError: 321878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10298.299878] LustreError: 321878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10299.523240] Lustre: server umount lustre-MDT0001 complete [10306.130303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10314.938366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10318.818296] LustreError: 321192:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006f8eefbf x1875743714868608/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10319.105169] LustreError: 323715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.169112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10319.234276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10323.965828] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10328.382389] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10328.397565] LustreError: 325475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10328.425257] LustreError: 325475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10332.492518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10333.082939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10338.140580] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10338.288669] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10345.974102] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10347.795169] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10353.881711] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10355.527904] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10363.876812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10363.880492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10363.885721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10363.888321] Lustre: Skipped 1 previous similar message [10363.916255] Lustre: Skipped 3 previous similar messages [10374.125353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10374.134628] Lustre: Skipped 3 previous similar messages [10375.136127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10375.318948] Lustre: server umount lustre-OST0000 complete [10384.353792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10384.357028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10384.390468] Lustre: Skipped 1 previous similar message [10385.723310] Lustre: server umount lustre-MDT0000 complete [10389.473622] LustreError: 325479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10389.501804] LustreError: 325479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10389.890822] LustreError: 321865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848792 with bad export cookie 18361756856440072547 [10389.893166] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10389.906384] LustreError: 321865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10390.384357] Lustre: server umount lustre-MDT0001 complete [10401.986432] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [10405.438972] Key type lgssc unregistered [10405.848673] LNet: 328151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10405.858589] LNetError: 328151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10405.871356] LNet: Removed LNI 192.168.204.115@tcp [10406.751756] Key type .llcrypt unregistered [10406.756145] Key type ._llcrypt unregistered [10432.238497] Key type ._llcrypt registered [10432.243473] Key type .llcrypt registered [10432.399762] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [10433.925641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10434.052884] alg: No test for adler32 (adler32-zlib) [10435.275812] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [10435.604343] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [10437.298822] Key type lgssc registered [10438.499298] Lustre: Echo OBD driver; http://www.lustre.org/ [10447.466806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10447.505698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.019308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10454.465992] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10458.596791] LustreError: 329566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10458.625257] LustreError: 329566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10463.714770] LustreError: 329565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10463.814198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.262385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10469.474394] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10478.566866] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10484.576804] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10489.714139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10489.735646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10489.735976] Lustre: Skipped 1 previous similar message [10489.750337] Lustre: Skipped 1 previous similar message [10493.907678] Lustre: server umount lustre-MDT0000 complete [10494.955173] LustreError: 329566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.977191] LustreError: 329566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10497.915553] LustreError: 329546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848900 with bad export cookie 18165336476852174343 [10497.920165] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10497.926674] LustreError: 329546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10498.420453] Lustre: server umount lustre-MDT0001 complete [10515.132296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.544992] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10515.591257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10515.737510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10520.840967] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10529.664955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10529.730738] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10529.758896] Lustre: 332152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10529.969581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10534.561270] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10542.106349] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10548.259808] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10557.870955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10558.124756] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10558.157890] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10558.166681] LustreError: 334036:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10558.179598] LustreError: 334036:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10558.190382] LustreError: 334036:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10558.326780] Lustre: server umount lustre-OST0000 complete [10558.333785] LustreError: 334036:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10568.245968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10568.432169] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10568.611805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10574.581326] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10578.133585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10582.235475] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10590.023089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10590.097400] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10590.167258] Lustre: lustre-OST0001: new disk, initializing [10590.172364] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10590.179151] Lustre: 335575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10590.264747] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10597.333521] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10599.536424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10599.546773] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10599.697224] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10604.922166] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10610.165264] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10610.165877] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.172533] LustreError: Skipped 1 previous similar message [10610.174528] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10610.187644] Lustre: Skipped 1 previous similar message [10614.144891] Lustre: server umount lustre-OST0001 complete [10615.272988] LustreError: 334504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10615.299613] LustreError: 334504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10619.363696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10619.374980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10619.398835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10619.414260] Lustre: Skipped 2 previous similar messages [10620.394330] LustreError: 334505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.418866] LustreError: 334505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10624.458945] Lustre: server umount lustre-OST0000 complete [10630.634565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10630.654040] Lustre: Skipped 1 previous similar message [10630.662640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10630.673513] Lustre: Skipped 3 previous similar messages [10635.289151] Lustre: server umount lustre-MDT0000 complete [10635.749703] LustreError: 333302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10635.773837] LustreError: 333302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10639.020334] LustreError: 332144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849041 with bad export cookie 18165336476852174896 [10639.024940] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10639.031889] LustreError: 332144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10639.308738] Lustre: server umount lustre-MDT0001 complete [10648.622643] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [10650.901811] Key type lgssc unregistered [10651.164978] LNet: 337536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10651.171293] LNetError: 337536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10651.201516] LNet: Removed LNI 192.168.204.115@tcp [10651.943476] Key type .llcrypt unregistered [10651.945892] Key type ._llcrypt unregistered [10720.735889] Key type ._llcrypt registered [10720.738045] Key type .llcrypt registered [10720.815633] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [10734.447641] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [10735.813478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10735.865886] alg: No test for adler32 (adler32-zlib) [10736.962755] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [10737.264444] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [10739.024299] Key type lgssc registered [10740.143410] Lustre: Echo OBD driver; http://www.lustre.org/ [10773.023159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10773.050580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10774.345860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10774.404720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10774.489447] Lustre: lustre-MDT0000: new disk, initializing [10774.586609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10774.614908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10778.888618] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10788.854153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10788.945614] Lustre: 342462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10788.999613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10789.002993] Lustre: Skipped 1 previous similar message [10789.098890] Lustre: lustre-MDT0001: new disk, initializing [10789.185460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10789.235323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10789.241152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10793.736124] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10802.619450] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10808.969726] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10815.937082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10816.242764] Lustre: lustre-OST0000: new disk, initializing [10816.248443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10816.260597] Lustre: 344431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10816.369920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10822.634827] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10825.816393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10825.825148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10825.922588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10831.471156] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10835.721544] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10835.970734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10840.416520] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10840.641619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10846.692941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10846.695700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10846.714830] Lustre: Skipped 1 previous similar message [10846.722485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10848.454916] Lustre: server umount lustre-OST0000 complete [10855.918024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10855.925993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10855.937796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10855.943818] Lustre: Skipped 1 previous similar message [10856.931517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10856.962306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10856.972133] Lustre: Skipped 1 previous similar message [10858.409551] Lustre: server umount lustre-MDT0000 complete [10862.177291] LustreError: 342457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849265 with bad export cookie 11159797774698971374 [10862.180527] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10862.184085] LustreError: 342457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10862.526419] Lustre: server umount lustre-MDT0001 complete [10871.310923] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:34:31 (1788849271) [10880.507617] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:34:40 (1788849280) [10888.560418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10888.931966] LustreError: 347187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10888.947147] LustreError: 347187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10889.002506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10892.879403] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10894.315154] LustreError: 347188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10898.408983] LustreError: 347187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10900.716294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10901.033916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10906.031857] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10913.900501] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10919.950729] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10926.320390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10926.687640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10932.016243] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10938.439701] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10948.831599] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10950.266495] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10951.929854] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10953.326158] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10955.133705] Lustre: Failing over lustre-MDT0000 [10955.399994] Lustre: server umount lustre-MDT0000 complete [10955.746028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10955.749857] LustreError: 348173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.764811] Lustre: Skipped 1 previous similar message [10955.786331] LustreError: 348173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10960.871757] LustreError: 347184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10960.886475] LustreError: 347184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10971.107643] LustreError: 348173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.133068] LustreError: 348173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10973.486882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10973.619530] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10973.847778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10973.897786] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10977.337649] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10977.625509] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10979.302100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10979.323189] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10979.373027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10985.430325] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10986.704205] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10989.905990] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10991.429091] Lustre: Failing over lustre-MDT0000 [10991.752695] Lustre: server umount lustre-MDT0000 complete [10992.714533] LustreError: 349679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10992.727180] LustreError: 349679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10994.659575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10994.664595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10994.682424] Lustre: Skipped 3 previous similar messages [11008.786581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11008.903218] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11009.086566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11009.164428] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11012.155970] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11012.819795] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11014.143091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11014.147992] Lustre: Skipped 2 previous similar messages [11014.178111] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11014.226598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11021.603432] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11023.044290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11029.474096] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11029.486786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11029.491841] Lustre: Skipped 2 previous similar messages [11029.492660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11034.595143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11034.604850] Lustre: Skipped 2 previous similar messages [11035.092808] Lustre: server umount lustre-MDT0000 complete [11038.669471] LustreError: 347921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849441 with bad export cookie 11159797774698974055 [11038.673827] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11038.680513] LustreError: 347921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11039.024644] Lustre: server umount lustre-MDT0001 complete [11053.090857] Lustre: server umount lustre-OST0000 complete [11068.194264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11068.621659] LustreError: 353524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.635969] LustreError: 353524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [11073.186899] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11081.056608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11081.438504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11081.443379] Lustre: Skipped 1 previous similar message [11085.315316] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11091.713389] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11096.403086] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11101.717655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11106.815144] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11113.039560] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11113.261744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11117.541120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11117.552087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11117.565263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11117.572708] Lustre: Skipped 2 previous similar messages [11122.448739] Lustre: server umount lustre-OST0000 complete [11127.780934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11127.784604] LustreError: Skipped 1 previous similar message [11131.425209] Lustre: server umount lustre-MDT0000 complete [11133.925738] LustreError: 355729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11133.948425] LustreError: 355729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11134.402469] LustreError: 353506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849537 with bad export cookie 11159797774698974797 [11134.412823] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11134.422426] LustreError: 353506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11134.667313] Lustre: server umount lustre-MDT0001 complete [11142.752319] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [11144.573691] Key type lgssc unregistered [11144.774622] LNet: 357053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11144.782823] LNetError: 357053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11144.797872] LNet: Removed LNI 192.168.204.115@tcp [11145.350158] Key type .llcrypt unregistered [11145.352958] Key type ._llcrypt unregistered [11165.605762] Key type ._llcrypt registered [11165.607656] Key type .llcrypt registered [11165.691222] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11166.271590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11166.295263] alg: No test for adler32 (adler32-zlib) [11167.328763] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [11167.579120] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [11169.264219] Key type lgssc registered [11170.120867] Lustre: Echo OBD driver; http://www.lustre.org/ [11176.414363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11176.432863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11177.773133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11180.925714] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11187.178945] LustreError: 358544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11187.191830] LustreError: 358544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11187.926537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11188.166398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11191.567512] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11198.096411] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11203.368656] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11209.228809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11209.519381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11214.974143] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11221.060913] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11222.826158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11238.373071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11238.375612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11238.385044] Lustre: Skipped 1 previous similar message [11238.393351] Lustre: Skipped 2 previous similar messages [11240.793405] Lustre: server umount lustre-MDT0000 complete [11243.494738] LustreError: 358543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11243.506568] LustreError: 358543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11243.843621] LustreError: 359279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849646 with bad export cookie 9528797019681182773 [11243.848122] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11243.853985] LustreError: 359279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11244.116454] Lustre: server umount lustre-MDT0001 complete [11257.739151] Lustre: server umount lustre-OST0000 complete [11268.488373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11268.805630] LustreError: 362276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11268.868614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11272.613475] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11274.210883] LustreError: 362277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.309438] LustreError: 362276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11279.709628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11280.073682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11283.723163] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11289.975332] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11294.468297] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11300.242699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11300.604928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11305.798275] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11311.872488] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11313.903317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11329.508801] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11329.511292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11329.522202] Lustre: Skipped 2 previous similar messages [11329.529497] Lustre: Skipped 2 previous similar messages [11333.186788] Lustre: server umount lustre-MDT0000 complete [11334.626818] LustreError: 362276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11334.648222] LustreError: 362276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11335.847118] LustreError: 362258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849738 with bad export cookie 9528797019681183886 [11335.849341] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11335.858886] LustreError: 362258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11336.095356] Lustre: server umount lustre-MDT0001 complete [11349.227547] Lustre: server umount lustre-OST0000 complete [11360.529501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.934865] LustreError: 366190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11360.995171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11364.529778] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11371.193753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11374.933212] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11380.939437] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11384.850201] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11389.807862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11394.893959] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11400.766235] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11405.492673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11410.921244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.935195] Lustre: Skipped 1 previous similar message [11410.945091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11410.946397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11410.960208] Lustre: Skipped 1 previous similar message [11411.335616] Lustre: server umount lustre-OST0000 complete [11416.037638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11416.041858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11416.052776] Lustre: Skipped 2 previous similar messages [11416.058945] Lustre: Skipped 1 previous similar message [11421.157211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11421.161182] Lustre: Skipped 1 previous similar message [11428.320136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11428.423266] Lustre: server umount lustre-MDT0000 complete [11430.375089] LustreError: 366191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.388602] LustreError: 366191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11431.088803] LustreError: 366171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849834 with bad export cookie 9528797019681184908 [11431.091761] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11431.107639] LustreError: 366171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11431.316519] Lustre: server umount lustre-MDT0001 complete [11438.554207] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [11440.578358] Key type lgssc unregistered [11440.766048] LNet: 369720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11440.773124] LNetError: 369720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11440.788921] LNet: Removed LNI 192.168.204.115@tcp [11441.435945] Key type .llcrypt unregistered [11441.443452] Key type ._llcrypt unregistered [11460.814823] Key type ._llcrypt registered [11460.818201] Key type .llcrypt registered [11460.937316] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11462.200121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11462.214496] alg: No test for adler32 (adler32-zlib) [11463.327190] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [11463.626955] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [11465.256963] Key type lgssc registered [11466.175900] Lustre: Echo OBD driver; http://www.lustre.org/ [11472.959799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11472.972901] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11472.982764] LustreError: 371136:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11472.999239] LustreError: 371136:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11473.007131] LustreError: 371136:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11473.014701] LustreError: 371136:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11473.020412] LustreError: 371136:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11479.391572] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11479.398641] LustreError: 371534:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11479.443636] LustreError: 371534:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11479.450903] LustreError: 371534:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11479.460706] LustreError: 371534:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11479.469480] LustreError: 371534:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11495.816159] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [11498.183249] Key type lgssc unregistered [11498.412936] LNet: 372535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11498.420818] LNetError: 372535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11498.448590] LNet: Removed LNI 192.168.204.115@tcp [11499.106844] Key type .llcrypt unregistered [11499.110080] Key type ._llcrypt unregistered [11513.415738] Key type ._llcrypt registered [11513.418241] Key type .llcrypt registered [11513.495464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [11526.770793] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11527.565165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11527.575815] alg: No test for adler32 (adler32-zlib) [11528.533562] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [11528.714897] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [11530.360522] Key type lgssc registered [11531.180781] Lustre: Echo OBD driver; http://www.lustre.org/ [11556.641989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11556.660466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11557.836408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11557.896769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11557.965973] Lustre: lustre-MDT0000: new disk, initializing [11558.040066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11558.062743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11561.341970] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11568.868016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11568.957869] Lustre: 375685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11568.996194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11568.999130] Lustre: Skipped 1 previous similar message [11569.056391] Lustre: lustre-MDT0001: new disk, initializing [11569.125926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11569.155180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11569.163769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11571.964043] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11579.143735] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11583.366511] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11588.262904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11588.504703] Lustre: lustre-OST0000: new disk, initializing [11588.509132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11588.516823] Lustre: 377656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11588.584675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11593.163341] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11594.791019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11594.801088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11594.877807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11600.107591] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11603.133112] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11603.278283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11606.549213] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11606.695215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11610.081161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11610.089483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11610.093367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11610.101489] Lustre: Skipped 1 previous similar message [11610.114735] Lustre: Skipped 1 previous similar message [11615.211100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11615.219072] Lustre: Skipped 1 previous similar message [11620.326531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11620.332905] Lustre: Skipped 1 previous similar message [11622.368168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11622.500241] Lustre: server umount lustre-OST0000 complete [11625.443666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11625.444582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11625.446229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11625.452549] Lustre: Skipped 1 previous similar message [11630.564588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11630.567341] Lustre: Skipped 2 previous similar messages [11631.645801] Lustre: server umount lustre-MDT0000 complete [11634.316509] LustreError: 375678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850037 with bad export cookie 15610352102154897061 [11634.319630] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11634.324026] LustreError: 375678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11634.616910] Lustre: server umount lustre-MDT0001 complete [11641.357824] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11642.720606] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:47:23 (1788850043) [11647.958402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11648.332333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11651.492735] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11653.602900] LustreError: 380114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11653.614259] LustreError: 380114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11658.326962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11658.615585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11662.035702] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11667.931396] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11672.384586] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11677.126411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11677.383717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11681.226679] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11686.249448] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11691.060761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11691.181186] Lustre: lustre-OST0001: new disk, initializing [11691.184224] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11691.188259] Lustre: 382952:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11695.326730] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11698.735264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11698.741112] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11698.793030] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11701.949408] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11708.495600] Lustre: lustre-OST0001: Not available for connect from 192.168.204.15@tcp (stopping) [11708.906987] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11708.909533] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11708.916207] LustreError: Skipped 1 previous similar message [11708.926740] Lustre: Skipped 1 previous similar message [11710.752962] Lustre: server umount lustre-OST0001 complete [11714.027056] LustreError: 381935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.037711] LustreError: 381935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11719.145514] LustreError: 381938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11719.161541] LustreError: 381938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11724.263312] LustreError: 381937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.275276] LustreError: 381937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11725.284412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11725.291407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11725.303677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11725.308295] Lustre: Skipped 2 previous similar messages [11729.387952] LustreError: 381937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.390976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11729.401229] LustreError: 381937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11731.256846] Lustre: server umount lustre-OST0000 complete [11748.320382] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11748.444283] Lustre: server umount lustre-MDT0000 complete [11749.857392] LustreError: 380114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11749.866255] LustreError: 380114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11750.935169] LustreError: 380094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850153 with bad export cookie 15610352102154897943 [11750.937746] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11750.941547] LustreError: 380094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11765.216150] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11765.405812] Lustre: server umount lustre-MDT0001 complete [11772.851913] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [11774.769772] Key type lgssc unregistered [11775.002559] LNet: 384937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11775.006653] LNetError: 384937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11775.019731] LNet: Removed LNI 192.168.204.115@tcp [11775.583963] Key type .llcrypt unregistered [11775.589393] Key type ._llcrypt unregistered [11806.344862] Key type ._llcrypt registered [11806.346960] Key type .llcrypt registered [11806.432731] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11807.065227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11807.091110] alg: No test for adler32 (adler32-zlib) [11808.046349] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [11808.234115] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [11809.888156] Key type lgssc registered [11810.566533] Lustre: Echo OBD driver; http://www.lustre.org/ [11815.518808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11815.538041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11816.688495] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11816.703359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11816.807075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11819.228909] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11824.317376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11824.373192] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11824.394752] Lustre: 387707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11824.568628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11826.945276] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11831.278950] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11834.657996] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11836.387603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.391132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11836.398962] Lustre: Skipped 1 previous similar message [11836.407206] Lustre: Skipped 1 previous similar message [11841.506295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11841.511426] Lustre: Skipped 1 previous similar message [11846.628484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11846.634604] Lustre: Skipped 1 previous similar message [11850.211159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11850.299512] Lustre: server umount lustre-MDT0000 complete [11852.383609] LustreError: 387698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850255 with bad export cookie 11626855306126713484 [11852.387094] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11852.392743] LustreError: 387698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11852.607769] Lustre: server umount lustre-MDT0001 complete [11858.204936] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:50:59 (1788850259) [11862.839900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11863.184764] LustreError: 390221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.193971] LustreError: 390221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11863.214685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11865.747657] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11868.643986] LustreError: 390222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11870.849186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11871.086700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11873.457888] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11877.609685] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11880.651682] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11884.315685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11884.411967] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11884.513099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11887.795693] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11892.170460] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11897.215472] Lustre: Permanently deactivating lustre-OST0000 [11897.228920] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11906.555577] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11907.508308] LustreError: 393299:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11907.518230] LustreError: 393299:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11907.525811] LustreError: 393299:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11908.857796] LustreError: 393347:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11908.861738] LustreError: 393347:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11916.143070] Lustre: server umount lustre-OST0000 complete [11917.789283] LustreError: 393598:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11917.797161] LustreError: 393598:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11922.401538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11922.402818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11922.403757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11922.416795] Lustre: Skipped 1 previous similar message [11927.522240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11927.527055] Lustre: Skipped 2 previous similar messages [11933.152159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11933.260705] Lustre: server umount lustre-MDT0000 complete [11935.517772] LustreError: 392054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850338 with bad export cookie 11626855306126714051 [11935.519942] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11935.527552] LustreError: 392054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11946.049297] Lustre: server umount lustre-MDT0001 complete [11951.603229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11954.261468] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11956.259788] LustreError: 394755:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11960.635536] Lustre: 395094:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.115@tcp [11962.041994] Lustre: server umount MGS complete [11969.258041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11969.569784] LustreError: 395817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11969.585659] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11969.588787] Lustre: Skipped 1 previous similar message [11969.631255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11972.456633] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11974.628171] LustreError: 395818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11977.692710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11977.858684] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11980.396914] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11984.773633] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11987.954563] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11989.040570] Lustre: Permanently reactivating lustre-OST0000 [11989.048723] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11989.054357] Lustre: Skipped 2 previous similar messages [12009.440127] LustreError: 397414:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12013.910842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12014.154170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12014.160544] Lustre: Skipped 1 previous similar message [12018.100564] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12022.806545] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12034.530127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12034.532629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12034.538122] LustreError: Skipped 1 previous similar message [12034.539065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12034.552232] Lustre: Skipped 1 previous similar message [12034.562893] Lustre: Skipped 3 previous similar messages [12037.310815] Lustre: server umount lustre-OST0000 complete [12044.772455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12044.776203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12044.788381] Lustre: Skipped 1 previous similar message [12045.957228] Lustre: server umount lustre-MDT0000 complete [12048.054874] LustreError: 395799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850450 with bad export cookie 11626855306126715094 [12048.060500] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12048.062731] LustreError: 395799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12048.264936] Lustre: server umount lustre-MDT0001 complete [12054.434439] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12056.073868] Key type lgssc unregistered [12056.270554] LNet: 400129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12056.282770] LNetError: 400129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12056.293546] LNet: Removed LNI 192.168.204.115@tcp [12056.783566] Key type .llcrypt unregistered [12056.785104] Key type ._llcrypt unregistered [12067.105155] Key type ._llcrypt registered [12067.107543] Key type .llcrypt registered [12067.172305] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [12077.345565] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12077.897552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12077.987954] alg: No test for adler32 (adler32-zlib) [12078.923752] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [12079.078094] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [12080.696227] Key type lgssc registered [12081.312864] Lustre: Echo OBD driver; http://www.lustre.org/ [12101.043830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12101.058043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12102.192749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12102.212973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12102.256557] Lustre: lustre-MDT0000: new disk, initializing [12102.303629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12102.316211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12104.890846] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12111.482830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12111.560408] Lustre: 403255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12111.595492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12111.600365] Lustre: Skipped 1 previous similar message [12111.648205] Lustre: lustre-MDT0001: new disk, initializing [12111.723150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12111.754717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12111.763929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12114.384985] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12120.119632] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12123.189779] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12126.928179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12127.053615] Lustre: lustre-OST0000: new disk, initializing [12127.055767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12127.058820] Lustre: 405221:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12127.101667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12127.349281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12127.358467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12127.399963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12130.191637] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12135.748705] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12138.099365] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12138.216679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12140.561738] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12140.675613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12143.077845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12143.081384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.082187] LustreError: Skipped 1 previous similar message [12143.083112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12143.089268] Lustre: Skipped 1 previous similar message [12148.197025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12148.202192] Lustre: Skipped 2 previous similar messages [12153.319667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12153.322169] Lustre: Skipped 1 previous similar message [12155.872333] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12155.953735] Lustre: server umount lustre-OST0000 complete [12157.923213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12157.929534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.940990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12163.554272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12163.557198] Lustre: Skipped 3 previous similar messages [12172.257293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12172.345807] Lustre: server umount lustre-MDT0000 complete [12174.102682] LustreError: 403248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850577 with bad export cookie 11219536529470743321 [12174.106265] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12174.109740] LustreError: 403248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12174.271121] Lustre: server umount lustre-MDT0001 complete [12178.483704] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12179.346802] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:56:20 (1788850580) [12182.534889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.762098] LustreError: 407683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12182.769715] LustreError: 407683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12182.794078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12184.869660] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12188.132584] LustreError: 407684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12188.922869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12189.120247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12191.271220] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12195.318762] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12198.274698] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12202.146683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12202.380719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12205.513821] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12209.584448] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12210.511278] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12212.292488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12222.947364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12222.954991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12222.963904] Lustre: Skipped 1 previous similar message [12222.967271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12222.971128] Lustre: Skipped 2 previous similar messages [12226.101949] Lustre: server umount lustre-OST0000 complete [12229.090131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12229.103176] Lustre: Skipped 2 previous similar messages [12239.330384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12239.334912] Lustre: Skipped 7 previous similar messages [12242.400190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12242.507558] Lustre: server umount lustre-MDT0000 complete [12244.450970] LustreError: 408755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12244.459265] LustreError: 408755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12244.496894] LustreError: 407664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850647 with bad export cookie 11219536529470744203 [12244.500277] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12244.504962] LustreError: 407664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12244.685846] Lustre: server umount lustre-MDT0001 complete [12250.000458] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12251.647109] Key type lgssc unregistered [12251.865545] LNet: 411404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12251.872939] LNetError: 411404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12251.891068] LNet: Removed LNI 192.168.204.115@tcp [12252.427705] Key type .llcrypt unregistered [12252.429293] Key type ._llcrypt unregistered [12284.262541] Key type ._llcrypt registered [12284.265121] Key type .llcrypt registered [12284.408702] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12285.097379] Key type .llcrypt unregistered [12285.100110] Key type ._llcrypt unregistered [12319.235711] Key type ._llcrypt registered [12319.241382] Key type .llcrypt registered [12319.480467] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12322.181304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12322.276965] alg: No test for adler32 (adler32-zlib) [12323.881636] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [12324.633891] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [12326.800937] Key type lgssc registered [12328.281302] Lustre: Echo OBD driver; http://www.lustre.org/ [12338.361659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12338.389669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12339.750248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12344.311926] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12349.924964] LustreError: 413616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12349.954378] LustreError: 413616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12352.974894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12353.273925] LustreError: 413611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.295775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12353.296529] LustreError: 413611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12358.626765] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.653858] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12359.360706] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12363.759463] LustreError: 413612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12363.790034] LustreError: 413612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12364.798078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12367.311875] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12369.891754] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12369.908360] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12374.774068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12375.062991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12380.091120] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12390.372578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12390.399964] LustreError: 414340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12390.405121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12390.505580] Lustre: server umount lustre-OST0000 complete [12395.101111] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12395.111828] LustreError: Skipped 1 previous similar message [12395.117667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12395.127360] Lustre: Skipped 1 previous similar message [12395.139022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12395.388431] Lustre: server umount lustre-MDT0000 complete [12399.327809] LustreError: 413595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850802 with bad export cookie 2355068010126029358 [12399.329593] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12399.343485] LustreError: 413595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12399.815394] Lustre: server umount lustre-MDT0001 complete [12410.228780] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12413.032105] Key type lgssc unregistered [12413.396783] LNet: 416760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12413.417103] LNetError: 416760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12413.431985] LNet: Removed LNI 192.168.204.115@tcp [12414.331748] Key type .llcrypt unregistered [12414.336184] Key type ._llcrypt unregistered [12447.788989] Key type ._llcrypt registered [12447.793157] Key type .llcrypt registered [12447.878242] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12449.166311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12449.186230] alg: No test for adler32 (adler32-zlib) [12450.263410] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [12450.593706] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [12452.328177] Key type lgssc registered [12453.463429] Lustre: Echo OBD driver; http://www.lustre.org/ [12461.936943] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12461.948243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12477.472635] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12482.592575] LustreError: 418159:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [12509.280219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12516.767683] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12525.740760] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12533.262191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12533.915138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12534.021920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12539.254566] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12539.366141] LustreError: 418184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12539.406039] LustreError: 418184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12543.476339] LustreError: 419199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12543.515737] LustreError: 419199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12548.580420] LustreError: 418184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.605206] LustreError: 418184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12548.825158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.416345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12554.513896] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12564.012989] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12570.739509] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12580.325737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12580.329894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12580.343848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12580.353633] Lustre: Skipped 1 previous similar message [12580.385251] Lustre: Skipped 1 previous similar message [12585.505429] Lustre: server umount lustre-OST0000 complete [12590.577138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12590.578770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12590.594902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12590.606489] LustreError: Skipped 1 previous similar message [12590.625364] LustreError: 419203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12590.733842] Lustre: server umount lustre-MDT0000 complete [12595.073273] LustreError: 418167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850997 with bad export cookie 7090076388957460072 [12595.077049] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12595.088391] LustreError: 418167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12595.510632] Lustre: server umount lustre-MDT0001 complete [12608.152287] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12610.886571] Key type lgssc unregistered [12611.113956] LNet: 421729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12611.123971] LNetError: 421729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12612.204960] LNet: Removed LNI 192.168.204.115@tcp [12613.211380] Key type .llcrypt unregistered [12613.214251] Key type ._llcrypt unregistered [12639.327290] Key type ._llcrypt registered [12639.329727] Key type .llcrypt registered [12639.462557] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12640.499117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12640.625888] alg: No test for adler32 (adler32-zlib) [12641.767318] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [12642.211485] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [12643.953684] Key type lgssc registered [12645.055720] Lustre: Echo OBD driver; http://www.lustre.org/ [12654.619183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12654.636552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12656.218287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12662.034655] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12666.338392] LustreError: 423143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12666.353410] LustreError: 423143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12670.932848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12671.413232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12676.010350] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12686.532684] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12694.028536] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12701.586796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12702.130279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12709.749592] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12719.153698] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12726.724196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12736.102996] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [12736.996297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12737.001399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12737.024829] Lustre: Skipped 1 previous similar message [12737.044019] Lustre: Skipped 2 previous similar messages [12740.430444] Lustre: server umount lustre-MDT0000 complete [12741.178312] LustreError: 423139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.217214] LustreError: 423124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.15@tcp arrived at 1788851146 with bad export cookie 6499157516128239992 [12745.822683] LustreError: 423125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851148 with bad export cookie 6499157516128239327 [12745.835674] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12745.851201] LustreError: 423125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12747.237804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12747.241020] LustreError: 423140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12747.253676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12747.253686] Lustre: Skipped 1 previous similar message [12747.261176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12747.272312] LustreError: 423140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12750.241409] LustreError: 423143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12750.242499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12750.270389] LustreError: 423143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12752.050144] Lustre: server umount lustre-OST0000 complete [12755.366841] LustreError: 423140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12755.392351] LustreError: 423140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12760.243540] Lustre: server umount lustre-MDT0001 complete [12773.664727] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12776.629871] Key type lgssc unregistered [12776.988822] LNet: 426851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12776.998311] LNetError: 426851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12777.036552] LNet: Removed LNI 192.168.204.115@tcp [12778.121404] Key type .llcrypt unregistered [12778.124097] Key type ._llcrypt unregistered [12806.335167] Key type ._llcrypt registered [12806.341242] Key type .llcrypt registered [12806.468638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12807.565156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12807.610445] alg: No test for adler32 (adler32-zlib) [12808.659861] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [12808.922506] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [12810.632224] Key type lgssc registered [12811.778735] Lustre: Echo OBD driver; http://www.lustre.org/ [12819.360214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12819.381191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12820.696638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12825.540460] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12830.194196] LustreError: 428264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12830.229911] LustreError: 428264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12833.406268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12833.743843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12838.233898] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12846.610729] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12853.665475] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12861.771353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12862.013119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12866.104739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12868.917211] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12877.354268] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12883.432882] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12883.456285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12883.469791] Lustre: Skipped 1 previous similar message [12884.589175] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [12887.961499] Lustre: server umount lustre-MDT0001 complete [12888.568480] LustreError: 428265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12888.596859] LustreError: 428265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12889.655367] LustreError: 428260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12893.683683] LustreError: 428260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12893.705884] LustreError: 428260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12896.738191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12896.750726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12896.775402] Lustre: Skipped 1 previous similar message [12896.789857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12898.789640] LustreError: 428992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12898.812382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12900.980129] Lustre: server umount lustre-OST0000 complete [12907.113906] Lustre: server umount lustre-MDT0000 complete [12921.653326] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12924.463429] Key type lgssc unregistered [12924.752510] LNet: 431948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12924.759038] LNetError: 431948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12924.774155] LNet: Removed LNI 192.168.204.115@tcp [12925.778727] Key type .llcrypt unregistered [12925.781253] Key type ._llcrypt unregistered [12955.124204] Key type ._llcrypt registered [12955.125950] Key type .llcrypt registered [12955.227173] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [12972.121056] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12973.069687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12973.291198] alg: No test for adler32 (adler32-zlib) [12974.418995] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [12974.704260] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [12976.344181] Key type lgssc registered [12977.541299] Lustre: Echo OBD driver; http://www.lustre.org/ [13023.639727] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [13043.813916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13043.850296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13045.261853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13045.332410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13045.500895] Lustre: lustre-MDT0000: new disk, initializing [13045.638708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13045.666759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13050.322681] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [13067.853630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13068.217644] Lustre: lustre-OST0000: new disk, initializing [13068.221547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13068.225830] Lustre: Skipped 1 previous similar message [13068.231838] Lustre: 436981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13068.327332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13075.821446] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [13078.152789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13078.161268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13078.185698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13086.935723] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13105.411064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13105.529196] Lustre: 435961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13105.613935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13105.720823] Lustre: lustre-MDT0001: new disk, initializing [13105.858452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13105.898996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13105.906298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13110.882363] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [13121.623312] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13121.973359] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13127.000446] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13127.279549] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13135.952417] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [13137.382264] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13137.391770] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13137.412645] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13137.422647] Lustre: Skipped 1 previous similar message [13137.435116] Lustre: Skipped 1 previous similar message [13139.182273] Lustre: server umount lustre-MDT0001 complete [13147.619325] LustreError: 435969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13147.642919] LustreError: 435969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13152.738541] LustreError: 435970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13152.761353] LustreError: 435970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13157.877103] LustreError: 435968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13157.908890] LustreError: 435968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13160.417970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13160.445790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13160.476064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13162.986200] LustreError: 439536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13162.990040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13166.070960] Lustre: server umount lustre-OST0000 complete [13168.102261] LustreError: 435974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13176.504588] Lustre: server umount lustre-MDT0000 complete [13188.767667] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [13191.258851] Key type lgssc unregistered [13191.459457] LNet: 440732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13191.465948] LNetError: 440732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13191.480257] LNet: Removed LNI 192.168.204.115@tcp [13192.118273] Key type .llcrypt unregistered [13192.120182] Key type ._llcrypt unregistered [13243.105815] Key type ._llcrypt registered [13243.119889] Key type .llcrypt registered [13243.307670] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [13243.753386] Key type .llcrypt unregistered [13243.761783] Key type ._llcrypt unregistered [13262.334388] Key type ._llcrypt registered [13262.336719] Key type .llcrypt registered [13262.439800] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [13263.618586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13263.636332] alg: No test for adler32 (adler32-zlib) [13264.918523] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [13265.210479] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [13266.912274] Key type lgssc registered [13268.341896] Lustre: Echo OBD driver; http://www.lustre.org/ [13302.369653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13302.395541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13303.667882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13303.696663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13303.790603] Lustre: lustre-MDT0000: new disk, initializing [13303.862068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13303.875407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13308.964533] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [13320.109369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13320.239516] Lustre: 444102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13320.298427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13320.304904] Lustre: Skipped 1 previous similar message [13320.405369] Lustre: lustre-MDT0001: new disk, initializing [13320.507924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13320.549649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13320.568375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13325.298330] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [13334.837703] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13341.750472] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13348.845267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13349.082441] Lustre: lustre-OST0000: new disk, initializing [13349.085772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13349.092549] Lustre: 446070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13349.170121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13355.725852] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [13356.613674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13356.631252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13356.762413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13366.172844] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13377.003304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13377.013886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13377.031969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13377.532371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13377.547308] Lustre: Skipped 1 previous similar message [13377.555719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13377.567938] Lustre: Skipped 2 previous similar messages [13379.504879] Lustre: server umount lustre-MDT0000 complete [13383.460750] LustreError: 444095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851786 with bad export cookie 8119965178790670494 [13383.474796] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13383.481367] LustreError: 444095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13383.868423] Lustre: server umount lustre-MDT0001 complete [13398.176496] Lustre: server umount lustre-OST0000 complete [13409.171734] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:16:50 (1788851810) [13417.721473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13418.156640] LustreError: 448101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13418.186721] LustreError: 448101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13418.305036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13423.520178] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [13423.585655] LustreError: 448102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13428.710587] LustreError: 448101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13432.891648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13433.442348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13438.502980] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [13446.575443] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13454.981207] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13477.297217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13477.532065] Lustre: Found index 0 for lustre-OST0000, updating log [13477.645660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13484.184277] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [13489.973086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13493.135540] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13500.393510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13500.401883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13500.414399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13500.441986] Lustre: Skipped 1 previous similar message [13505.752276] Lustre: server umount lustre-OST0000 complete [13521.935455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13521.952671] Lustre: Skipped 1 previous similar message [13521.965818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13521.973351] Lustre: Skipped 3 previous similar messages [13527.009283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13527.021169] Lustre: Skipped 1 previous similar message [13536.224159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13536.351045] Lustre: server umount lustre-MDT0000 complete [13539.321828] LustreError: 449953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13539.366875] LustreError: 449953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13540.174868] LustreError: 448082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851943 with bad export cookie 8119965178790671887 [13540.184305] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13540.184822] LustreError: 448082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13540.465677] Lustre: server umount lustre-MDT0001 complete [13542.752366] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [13549.137969] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:19:09 (1788851949) [14176.148308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14176.695661] LustreError: 454514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14176.782620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14181.449726] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [14181.860900] LustreError: 454515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14186.985184] LustreError: 454514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14192.186654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14192.622394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14197.553168] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [14207.011940] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14213.935890] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14233.953978] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [14236.013423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14236.265306] Lustre: Found index 0 for lustre-OST0000, updating log [14236.638721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14243.513551] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [14252.802199] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14256.103551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14256.114667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14256.137587] Lustre: Skipped 1 previous similar message [14256.144778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14256.153345] Lustre: Skipped 6 previous similar messages [14261.654584] Lustre: server umount lustre-OST0000 complete [14266.338323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14266.372581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14266.387147] Lustre: Skipped 1 previous similar message [14268.453678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14268.465116] Lustre: Skipped 3 previous similar messages [14271.771539] Lustre: server umount lustre-MDT0000 complete [14272.548405] LustreError: 455244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14272.588202] LustreError: 455244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14276.645149] LustreError: 454510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14276.683193] LustreError: 454510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [14277.208659] LustreError: 454495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852680 with bad export cookie 8119965178790672972 [14277.235927] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14277.813576] Lustre: server umount lustre-MDT0001 complete [14293.344231] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [14320.281946] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [14333.186910] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [14379.705132] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:33:00 (1788852780) [14404.069703] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [14407.440698] Key type lgssc unregistered [14407.865885] LNet: 461130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14407.876367] LNetError: 461130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14497.001624] LNet: Removed LNI 192.168.204.115@tcp [14498.196705] Key type .llcrypt unregistered [14498.198519] Key type ._llcrypt unregistered [14518.812655] Key type ._llcrypt registered [14518.819286] Key type .llcrypt registered [14519.039612] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [14521.013153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14521.140096] alg: No test for adler32 (adler32-zlib) [14522.232957] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [14522.530722] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [14524.265976] Key type lgssc registered [14525.626045] Lustre: Echo OBD driver; http://www.lustre.org/ [14530.931412] LNet: Added LNI 192.168.204.115@tcp100 [8/256/0/180] [14531.904553] LNet: Added LNI 192.168.204.115@tcp101 [8/256/0/180] [14533.144496] LNet: Added LNI 192.168.204.115@tcp102 [8/256/0/180] [14534.168106] LNet: Added LNI 192.168.204.115@tcp103 [8/256/0/180] [14535.344365] LNet: Added LNI 192.168.204.115@tcp104 [8/256/0/180] [14536.448568] LNet: Added LNI 192.168.204.115@tcp105 [8/256/0/180] [14537.480642] LNet: Added LNI 192.168.204.115@tcp106 [8/256/0/180] [14538.684253] LNet: Added LNI 192.168.204.115@tcp107 [8/256/0/180] [14540.053647] LNet: Added LNI 192.168.204.115@tcp108 [8/256/0/180] [14541.751366] LNet: Added LNI 192.168.204.115@tcp109 [8/256/0/180] [14543.273956] LNet: Added LNI 192.168.204.115@tcp110 [8/256/0/180] [14544.565306] LNet: Added LNI 192.168.204.115@tcp111 [8/256/0/180] [14546.198077] LNet: Added LNI 192.168.204.115@tcp112 [8/256/0/180] [14547.715069] LNet: Added LNI 192.168.204.115@tcp113 [8/256/0/180] [14549.055645] LNet: Added LNI 192.168.204.115@tcp114 [8/256/0/180] [14550.526255] LNet: Added LNI 192.168.204.115@tcp115 [8/256/0/180] [14552.328424] LNet: Added LNI 192.168.204.115@tcp116 [8/256/0/180] [14554.156119] LNet: Added LNI 192.168.204.115@tcp117 [8/256/0/180] [14555.424772] LNet: Added LNI 192.168.204.115@tcp118 [8/256/0/180] [14556.378966] LNet: Added LNI 192.168.204.115@tcp119 [8/256/0/180] [14557.510643] LNet: Added LNI 192.168.204.115@tcp120 [8/256/0/180] [14558.961779] LNet: Added LNI 192.168.204.115@tcp121 [8/256/0/180] [14560.637268] LNet: Added LNI 192.168.204.115@tcp122 [8/256/0/180] [14561.933117] LNet: Added LNI 192.168.204.115@tcp123 [8/256/0/180] [14563.272152] LNet: Added LNI 192.168.204.115@tcp124 [8/256/0/180] [14565.015667] LNet: Added LNI 192.168.204.115@tcp125 [8/256/0/180] [14566.240237] LNet: Added LNI 192.168.204.115@tcp126 [8/256/0/180] [14567.678085] LNet: Added LNI 192.168.204.115@tcp127 [8/256/0/180] [14568.754615] LNet: Added LNI 192.168.204.115@tcp128 [8/256/0/180] [14569.639446] LNet: Added LNI 192.168.204.115@tcp129 [8/256/0/180] [14570.657316] LNet: Added LNI 192.168.204.115@tcp130 [8/256/0/180] [14571.718739] LNet: Added LNI 192.168.204.115@tcp131 [8/256/0/180] [14572.654948] LNet: Added LNI 192.168.204.115@tcp132 [8/256/0/180] [14573.659102] LNet: Added LNI 192.168.204.115@tcp133 [8/256/0/180] [14574.577313] LNet: Added LNI 192.168.204.115@tcp134 [8/256/0/180] [14575.646628] LNet: Added LNI 192.168.204.115@tcp135 [8/256/0/180] [14584.903257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14584.948945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14586.080753] Lustre: 464157:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [14586.146108] LNetError: 461612:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [14586.165977] LNetError: Protocol error connecting to 192.168.204.115@tcp121 at host 192.168.204.115:988: is it running a compatible version of Lustre? [14586.191449] LNetError: Skipped 7 previous similar messages [14586.331440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14586.371436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14586.511618] Lustre: lustre-MDT0000: new disk, initializing [14586.599089] Lustre: 464214:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [14586.611298] Lustre: 464214:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14586.642934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14586.674675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14587.105285] LNet: 4 local NIs in recovery (showing 4): 192.168.204.115@tcp129, 192.168.204.115@tcp128, 192.168.204.115@tcp127, 192.168.204.115@tcp121 [14587.108996] LNetError: 464177:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [14587.146440] LNetError: 464177:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14587.157747] LNetError: Protocol error connecting to 192.168.204.115@tcp100 at host 192.168.204.115:988: is it running a compatible version of Lustre? [14587.178249] LNetError: Skipped 3 previous similar messages [14593.023767] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [14606.875994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14607.120587] Lustre: 464173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14607.231170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14607.235678] Lustre: Skipped 1 previous similar message [14607.363531] Lustre: lustre-MDT0001: new disk, initializing [14607.438157] Lustre: 464977:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [14607.456790] Lustre: 464977:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14607.722324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14607.843985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14607.876970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14613.042713] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [14615.603215] Lustre: 465341:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [14615.615433] Lustre: 465341:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14621.866548] LNetError: 464185:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [14621.897919] LNetError: 464185:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14621.926838] LNetError: Protocol error connecting to 192.168.204.115@tcp123 at host 192.168.204.115:988: is it running a compatible version of Lustre? [14623.907477] LNetError: 464179:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [14623.920353] LNetError: 464179:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14624.284975] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14624.929941] LNetError: Protocol error connecting to 192.168.204.115@tcp121 at host 192.168.204.115:988: is it running a compatible version of Lustre? [14624.941200] LNetError: Skipped 2 previous similar messages [14628.013889] LNetError: 464183:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [14628.041582] LNetError: 464183:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14631.500644] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14633.126201] LNetError: Protocol error connecting to 192.168.204.115@tcp123 at host 192.168.204.115:988: is it running a compatible version of Lustre? [14633.142583] LNetError: Skipped 3 previous similar messages [14636.199525] LNetError: 464178:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [14636.207939] LNetError: 464178:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14639.625221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14639.951242] Lustre: lustre-OST0000: new disk, initializing [14639.962695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14639.970484] Lustre: 466152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14640.076169] Lustre: 466157:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [14640.086153] Lustre: 466157:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14640.150059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14646.442760] LNetError: Protocol error connecting to 192.168.204.115@tcp123 at host 192.168.204.115:988: is it running a compatible version of Lustre? [14646.461461] LNetError: Skipped 5 previous similar messages [14647.169249] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [14648.855713] Lustre: 466588:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [14648.860461] Lustre: 466588:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14648.923911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14648.951754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14649.133543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14652.579545] LNetError: 464185:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [14652.588862] LNetError: 464185:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14657.582071] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14664.867272] LNetError: Protocol error connecting to 192.168.204.115@tcp124 at host 192.168.204.115:988: is it running a compatible version of Lustre? [14664.883734] LNetError: Skipped 8 previous similar messages [14669.803585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14669.824392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14669.863033] Lustre: Skipped 1 previous similar message [14669.913909] Lustre: Skipped 1 previous similar message [14673.352435] Lustre: server umount lustre-OST0000 complete [14679.522777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14679.535521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14679.550290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14685.163141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14685.178639] Lustre: Skipped 3 previous similar messages [14685.346626] LNetError: 464181:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [14685.371730] LNetError: 464181:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14689.249603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14689.261552] Lustre: Skipped 1 previous similar message [14693.344109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14693.672544] Lustre: server umount lustre-MDT0000 complete [14697.810787] LustreError: 464166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853100 with bad export cookie 14863420328250458264 [14697.819161] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14697.826926] LustreError: 464166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14698.268831] Lustre: server umount lustre-MDT0001 complete [14710.562796] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [14713.834500] Key type lgssc unregistered [14714.018264] LNetError: Protocol error connecting to 192.168.204.115@tcp100 at host 192.168.204.115:988: is it running a compatible version of Lustre? [14714.027107] LNetError: Skipped 7 previous similar messages [14714.422125] LNet: 468067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14714.439077] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14714.446562] LNet: Removed LNI 192.168.204.115@tcp [14715.488275] LNet: Removed LNI 192.168.204.115@tcp100 [14715.493842] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14715.500287] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14715.507728] LNet: Removed LNI 192.168.204.115@tcp101 [14716.512205] LNet: Removed LNI 192.168.204.115@tcp102 [14716.520437] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14716.534581] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14716.548648] LNet: Removed LNI 192.168.204.115@tcp103 [14717.600343] LNet: Removed LNI 192.168.204.115@tcp104 [14718.626158] LNet: Removed LNI 192.168.204.115@tcp105 [14718.635829] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14718.642374] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14718.660706] LNet: Removed LNI 192.168.204.115@tcp106 [14719.712175] LNet: Removed LNI 192.168.204.115@tcp107 [14720.736292] LNet: Removed LNI 192.168.204.115@tcp108 [14721.824839] LNet: Removed LNI 192.168.204.115@tcp109 [14722.848170] LNet: Removed LNI 192.168.204.115@tcp110 [14722.851522] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14722.866679] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14722.879190] LNet: Removed LNI 192.168.204.115@tcp111 [14723.936177] LNet: Removed LNI 192.168.204.115@tcp112 [14724.960217] LNet: Removed LNI 192.168.204.115@tcp113 [14725.984397] LNet: Removed LNI 192.168.204.115@tcp114 [14727.008215] LNet: Removed LNI 192.168.204.115@tcp115 [14728.032147] LNet: Removed LNI 192.168.204.115@tcp116 [14729.056517] LNet: Removed LNI 192.168.204.115@tcp117 [14730.081171] LNet: Removed LNI 192.168.204.115@tcp118 [14731.104170] LNet: Removed LNI 192.168.204.115@tcp119 [14731.113083] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14731.120103] LNetError: 468067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14731.128425] LNet: Removed LNI 192.168.204.115@tcp120 [14732.192147] LNet: Removed LNI 192.168.204.115@tcp121 [14733.216337] LNet: Removed LNI 192.168.204.115@tcp122 [14734.240260] LNet: Removed LNI 192.168.204.115@tcp123 [14735.264157] LNet: Removed LNI 192.168.204.115@tcp124 [14736.296686] LNet: Removed LNI 192.168.204.115@tcp125 [14737.312156] LNet: Removed LNI 192.168.204.115@tcp126 [14738.338776] LNet: Removed LNI 192.168.204.115@tcp127 [14739.360254] LNet: Removed LNI 192.168.204.115@tcp128 [14740.384161] LNet: Removed LNI 192.168.204.115@tcp129 [14740.386197] LNet: Removed LNI 192.168.204.115@tcp130 [14740.400505] LNet: Removed LNI 192.168.204.115@tcp131 [14740.408419] LNet: Removed LNI 192.168.204.115@tcp132 [14740.414217] LNet: Removed LNI 192.168.204.115@tcp133 [14740.421934] LNet: Removed LNI 192.168.204.115@tcp134 [14740.436348] LNet: Removed LNI 192.168.204.115@tcp135 [14741.308927] Key type .llcrypt unregistered [14741.312067] Key type ._llcrypt unregistered [14768.010199] Key type ._llcrypt registered [14768.013944] Key type .llcrypt registered [14768.175778] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [14787.007718] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [14788.045184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14788.067652] alg: No test for adler32 (adler32-zlib) [14789.324854] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [14789.542588] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [14791.200213] Key type lgssc registered [14792.139796] Lustre: Echo OBD driver; http://www.lustre.org/ [14836.319872] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:40:35 (1788853235) [14859.169744] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [14862.310289] Key type lgssc unregistered [14862.588243] LNet: 472156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14862.600570] LNetError: 472156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14862.628397] LNet: Removed LNI 192.168.204.115@tcp [14863.389431] Key type .llcrypt unregistered [14863.395069] Key type ._llcrypt unregistered [14881.806920] Key type ._llcrypt registered [14881.810272] Key type .llcrypt registered [14881.981883] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [14883.498073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14883.597144] alg: No test for adler32 (adler32-zlib) [14884.694603] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [14884.991888] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [14886.752267] Key type lgssc registered [14888.346077] Lustre: Echo OBD driver; http://www.lustre.org/ [14900.233467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14900.269730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14901.644366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14901.679597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14901.783720] Lustre: lustre-MDT0000: new disk, initializing [14901.889191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14901.914842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14906.856026] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [14917.705192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14917.849593] Lustre: 473363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14917.906393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14917.910891] Lustre: Skipped 1 previous similar message [14917.978821] Lustre: lustre-MDT0001: new disk, initializing [14918.064094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14918.097348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14918.108600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14922.919685] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [14935.019700] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14941.934106] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14950.719688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14951.088602] Lustre: lustre-OST0000: new disk, initializing [14951.101714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14951.113848] Lustre: 475331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14951.207247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14958.706128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14958.725414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14958.919592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14959.437877] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [14976.948962] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14983.274981] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [14991.070848] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14991.153260] LNet: Added LNI 192.168.204.115@tcp10 [8/256/0/180] [14998.350739] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14998.398566] LNet: Added LNI 192.168.204.115@tcp11 [8/256/0/180] [15002.795757] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15002.837713] LNet: Added LNI 192.168.204.115@tcp12 [8/256/0/180] [15007.894643] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15007.935836] LNet: Added LNI 192.168.204.115@tcp13 [8/256/0/180] [15014.718788] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15014.772046] LNet: Added LNI 192.168.204.115@tcp14 [8/256/0/180] [15021.129229] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15021.165846] LNet: Added LNI 192.168.204.115@tcp15 [8/256/0/180] [15027.943638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15028.013334] LNet: Added LNI 192.168.204.115@tcp16 [8/256/0/180] [15033.390333] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15033.472524] LNet: Added LNI 192.168.204.115@tcp17 [8/256/0/180] [15038.593861] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15038.616294] LNet: Added LNI 192.168.204.115@tcp18 [8/256/0/180] [15044.976077] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15045.044657] LNet: Added LNI 192.168.204.115@tcp19 [8/256/0/180] [15052.595213] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15052.635766] LNet: Added LNI 192.168.204.115@tcp20 [8/256/0/180] [15060.290628] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15060.318403] LNet: Added LNI 192.168.204.115@tcp21 [8/256/0/180] [15067.358995] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15067.388576] LNet: Added LNI 192.168.204.115@tcp22 [8/256/0/180] [15074.484543] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15074.512222] LNet: Added LNI 192.168.204.115@tcp23 [8/256/0/180] [15081.405242] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15081.434509] LNet: Added LNI 192.168.204.115@tcp24 [8/256/0/180] [15088.463377] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15088.501244] LNet: Added LNI 192.168.204.115@tcp25 [8/256/0/180] [15096.088259] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15096.124384] LNet: Added LNI 192.168.204.115@tcp26 [8/256/0/180] [15104.173514] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15104.228421] LNet: Added LNI 192.168.204.115@tcp27 [8/256/0/180] [15112.662874] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15112.701252] LNet: Added LNI 192.168.204.115@tcp28 [8/256/0/180] [15120.954596] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15120.987192] LNet: Added LNI 192.168.204.115@tcp29 [8/256/0/180] [15127.671194] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15127.703415] LNet: Added LNI 192.168.204.115@tcp30 [8/256/0/180] [15136.173225] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15136.197680] LNet: Added LNI 192.168.204.115@tcp31 [8/256/0/180] [15144.247842] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15144.288941] LNet: Added LNI 192.168.204.115@tcp32 [8/256/0/180] [15151.307304] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15151.357908] LNet: Added LNI 192.168.204.115@tcp33 [8/256/0/180] [15157.456220] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15157.482210] LNet: Added LNI 192.168.204.115@tcp34 [8/256/0/180] [15164.995373] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15165.025296] LNet: Added LNI 192.168.204.115@tcp35 [8/256/0/180] [15172.186167] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15172.202996] LNet: Added LNI 192.168.204.115@tcp36 [8/256/0/180] [15180.743948] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15180.803885] LNet: Added LNI 192.168.204.115@tcp37 [8/256/0/180] [15189.067166] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15189.094936] LNet: Added LNI 192.168.204.115@tcp38 [8/256/0/180] [15199.517977] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15199.582806] LNet: Added LNI 192.168.204.115@tcp39 [8/256/0/180] [15207.027721] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15207.076916] LNet: Added LNI 192.168.204.115@tcp40 [8/256/0/180] [15218.206427] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15218.241977] LNet: Added LNI 192.168.204.115@tcp41 [8/256/0/180] [15227.240981] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15227.263039] LNet: Added LNI 192.168.204.115@tcp42 [8/256/0/180] [15235.739496] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15235.796336] LNet: Added LNI 192.168.204.115@tcp43 [8/256/0/180] [15244.436453] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15244.478110] LNet: Added LNI 192.168.204.115@tcp44 [8/256/0/180] [15252.379747] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15252.421260] LNet: Added LNI 192.168.204.115@tcp45 [8/256/0/180] [15262.244159] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15262.275957] LNet: Added LNI 192.168.204.115@tcp46 [8/256/0/180] [15275.394721] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15275.443418] LNet: Added LNI 192.168.204.115@tcp47 [8/256/0/180] [15286.632206] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15286.657927] LNet: Added LNI 192.168.204.115@tcp48 [8/256/0/180] [15296.052640] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15296.078542] LNet: Added LNI 192.168.204.115@tcp49 [8/256/0/180] [15306.121870] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15306.144568] LNet: Added LNI 192.168.204.115@tcp50 [8/256/0/180] [15337.964351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15337.973268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15337.986119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15338.007759] Lustre: Skipped 2 previous similar messages [15342.404287] Lustre: server umount lustre-MDT0000 complete [15346.421482] LustreError: 473355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853749 with bad export cookie 17647205161587337296 [15346.428950] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15346.433957] LustreError: 473355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15346.657792] Lustre: server umount lustre-MDT0001 complete [15364.577202] Lustre: 472634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853751/real 1788853751] req@00000000d26886a4 x1875748624460416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788853767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15364.615828] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15367.648887] Lustre: 472636:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853754/real 1788853754] req@00000000f690c58f x1875748624460544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788853770 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15369.696346] Lustre: 472637:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853756/real 1788853756] req@00000000bb15b308 x1875748624460672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788853772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15372.772786] Lustre: 472635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853759/real 1788853759] req@000000005084a5f7 x1875748624460928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788853775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15373.983291] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [15377.888165] Lustre: 472634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853764/real 1788853764] req@000000005a5f0756 x1875748624461056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788853780 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15386.140315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15391.332812] LustreError: 472633:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b79bfe54 x1875748624461824/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15391.659586] LustreError: 475325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15391.668847] LustreError: 475325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15391.708190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15393.763547] LustreError: 475326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15393.765501] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15394.791812] LustreError: 475325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15396.140098] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15399.906939] LustreError: 475326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15405.036240] LustreError: 475325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15405.726910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15406.132074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15409.202734] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15411.997058] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15420.948326] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15428.104575] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15455.211215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15455.220897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15455.228646] Lustre: Skipped 3 previous similar messages [15460.138469] Lustre: server umount lustre-OST0000 complete [15465.441782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15465.451658] Lustre: Skipped 1 previous similar message [15465.455279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15465.468126] Lustre: Skipped 1 previous similar message [15470.564197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15470.577631] Lustre: Skipped 1 previous similar message [15475.683527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15475.700690] Lustre: Skipped 1 previous similar message [15478.241483] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15478.531709] Lustre: server umount lustre-MDT0000 complete [15480.813081] LustreError: 499601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15480.841196] LustreError: 499601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15483.183784] LustreError: 473355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853886 with bad export cookie 17647205161587339970 [15483.186026] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15483.197657] LustreError: 473355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15483.676448] Lustre: server umount lustre-MDT0001 complete [15497.239503] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [15500.409499] Key type lgssc unregistered [15500.919241] LNet: 502406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15500.937193] LNetError: 502406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15500.948247] LNet: Removed LNI 192.168.204.115@tcp [15500.949775] LNet: Removed LNI 192.168.204.115@tcp10 [15500.958975] LNet: Removed LNI 192.168.204.115@tcp11 [15500.969844] LNet: Removed LNI 192.168.204.115@tcp12 [15500.976806] LNet: Removed LNI 192.168.204.115@tcp13 [15500.978819] LNet: Removed LNI 192.168.204.115@tcp14 [15500.985668] LNet: Removed LNI 192.168.204.115@tcp15 [15500.990803] LNet: Removed LNI 192.168.204.115@tcp16 [15500.995900] LNet: Removed LNI 192.168.204.115@tcp17 [15501.002653] LNet: Removed LNI 192.168.204.115@tcp18 [15501.009588] LNet: Removed LNI 192.168.204.115@tcp19 [15501.013432] LNet: Removed LNI 192.168.204.115@tcp20 [15501.018430] LNet: Removed LNI 192.168.204.115@tcp21 [15501.021375] LNet: Removed LNI 192.168.204.115@tcp22 [15501.024885] LNet: Removed LNI 192.168.204.115@tcp23 [15501.027387] LNet: Removed LNI 192.168.204.115@tcp24 [15501.029386] LNet: Removed LNI 192.168.204.115@tcp25 [15501.034540] LNet: Removed LNI 192.168.204.115@tcp26 [15501.041133] LNet: Removed LNI 192.168.204.115@tcp27 [15501.044805] LNet: Removed LNI 192.168.204.115@tcp28 [15501.047448] LNet: Removed LNI 192.168.204.115@tcp29 [15501.049387] LNet: Removed LNI 192.168.204.115@tcp30 [15501.052954] LNet: Removed LNI 192.168.204.115@tcp31 [15501.056490] LNet: Removed LNI 192.168.204.115@tcp32 [15501.058692] LNet: Removed LNI 192.168.204.115@tcp33 [15501.061735] LNet: Removed LNI 192.168.204.115@tcp34 [15501.064427] LNet: Removed LNI 192.168.204.115@tcp35 [15501.066423] LNet: Removed LNI 192.168.204.115@tcp36 [15501.068674] LNet: Removed LNI 192.168.204.115@tcp37 [15501.071331] LNet: Removed LNI 192.168.204.115@tcp38 [15501.073728] LNet: Removed LNI 192.168.204.115@tcp39 [15501.075755] LNet: Removed LNI 192.168.204.115@tcp40 [15501.077652] LNet: Removed LNI 192.168.204.115@tcp41 [15501.079486] LNet: Removed LNI 192.168.204.115@tcp42 [15501.081365] LNet: Removed LNI 192.168.204.115@tcp43 [15501.083140] LNet: Removed LNI 192.168.204.115@tcp44 [15501.085346] LNet: Removed LNI 192.168.204.115@tcp45 [15501.087689] LNet: Removed LNI 192.168.204.115@tcp46 [15501.090133] LNet: Removed LNI 192.168.204.115@tcp47 [15501.092600] LNet: Removed LNI 192.168.204.115@tcp48 [15501.099393] LNet: Removed LNI 192.168.204.115@tcp49 [15501.111170] LNet: Removed LNI 192.168.204.115@tcp50 [15501.975771] Key type .llcrypt unregistered [15501.978860] Key type ._llcrypt unregistered [15526.938487] Key type ._llcrypt registered [15526.941776] Key type .llcrypt registered [15527.088171] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [15546.482660] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [15547.764795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15547.995499] alg: No test for adler32 (adler32-zlib) [15549.116982] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [15549.430360] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [15551.241092] Key type lgssc registered [15552.224351] Lustre: Echo OBD driver; http://www.lustre.org/ [15598.417582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15598.450496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15599.766184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15599.805406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15599.877332] Lustre: lustre-MDT0000: new disk, initializing [15599.987431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15600.004883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15604.692904] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15617.588658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15617.791351] Lustre: 505607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15617.897397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15617.901140] Lustre: Skipped 1 previous similar message [15617.963420] Lustre: lustre-MDT0001: new disk, initializing [15618.070416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15618.145994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15618.162594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15625.750887] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15640.993390] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15651.011124] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15661.283269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15661.499308] Lustre: *** cfs_fail_loc=90f, val=0*** [15661.566713] Lustre: lustre-OST0000: new disk, initializing [15661.571953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15661.577643] Lustre: 507626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15661.659222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15668.340890] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15669.887386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15669.899928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15670.041281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15678.356819] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15690.729498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15690.754169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15690.776778] Lustre: Skipped 1 previous similar message [15690.786455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15690.795312] Lustre: Skipped 1 previous similar message [15695.865317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15695.875224] Lustre: Skipped 1 previous similar message [15696.384945] Lustre: server umount lustre-OST0000 complete [15705.058211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15705.089247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15705.124652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15711.202144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15711.207245] Lustre: Skipped 3 previous similar messages [15716.322027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15716.366567] Lustre: Skipped 1 previous similar message [15718.369282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15718.835781] Lustre: server umount lustre-MDT0000 complete [15724.019494] LustreError: 506403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854126 with bad export cookie 5748429303960597435 [15724.032504] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15724.047853] LustreError: 506403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15724.584896] Lustre: server umount lustre-MDT0001 complete [15737.935814] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [15740.995134] Key type lgssc unregistered [15741.405073] LNet: 509534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15741.430122] LNetError: 509534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15741.466693] LNet: Removed LNI 192.168.204.115@tcp [15742.584145] Key type .llcrypt unregistered [15742.585831] Key type ._llcrypt unregistered [15771.971247] Key type ._llcrypt registered [15771.973075] Key type .llcrypt registered [15772.044763] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [15773.040703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15773.069336] alg: No test for adler32 (adler32-zlib) [15774.475265] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [15774.971153] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [15776.800344] Key type lgssc registered [15778.720544] Lustre: Echo OBD driver; http://www.lustre.org/ [15788.313643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15788.350236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15790.025317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15795.299943] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15799.270191] LustreError: 510803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15799.301972] LustreError: 510803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15804.392094] LustreError: 510804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15804.549685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15804.949441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15808.451909] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15818.389097] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15826.105799] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15835.743177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15836.125893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15843.442530] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15851.779792] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15875.423619] loop0: detected capacity change from 0 to 204800000 [15875.718210] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15875.727425] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15875.739727] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15883.096356] loop0: detected capacity change from 0 to 204800000 [15883.236233] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15883.248926] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15883.262294] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15889.096486] loop0: detected capacity change from 0 to 204800000 [15897.059806] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15897.503830] Lustre: fs15246-OST0000: new disk, initializing [15897.513132] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15897.526731] Lustre: 514422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15897.634203] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15907.275661] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15921.075238] loop1: detected capacity change from 0 to 204800000 [15930.329541] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15930.471209] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15930.553780] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15930.789856] Lustre: fs15246-MDT0000: new disk, initializing [15931.248177] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15931.318486] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15931.468607] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15932.516747] Lustre: 515547:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000bb15b308 x1875749557557376/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788854351 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15936.986169] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [15941.748267] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15941.765897] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15941.822293] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15969.263963] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15969.274887] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15971.710799] Lustre: server umount fs15246-MDT0000 complete [15974.370959] LustreError: 510800:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15979.495614] LustreError: 510799:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15982.812966] Lustre: server umount fs15246-OST0000 complete [15997.922415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15997.925726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15997.940371] LustreError: Skipped 1 previous similar message [15997.947455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15997.967350] Lustre: Skipped 1 previous similar message [15997.991242] Lustre: Skipped 1 previous similar message [15999.994315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16000.003714] Lustre: Skipped 1 previous similar message [16002.346479] Lustre: server umount lustre-OST0000 complete [16009.699546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16009.711446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16009.749201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16013.997429] Lustre: server umount lustre-MDT0000 complete [16015.345329] LustreError: 510804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16015.366040] LustreError: 510804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16018.759776] LustreError: 510785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854421 with bad export cookie 2592989553069604014 [16018.779832] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16018.781605] LustreError: 510785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16019.275323] Lustre: server umount lustre-MDT0001 complete [16030.588514] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [16033.297799] Key type lgssc unregistered [16033.547056] LNet: 518643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16033.557564] LNetError: 518643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16033.581264] LNet: Removed LNI 192.168.204.115@tcp [16034.531739] Key type .llcrypt unregistered [16034.535377] Key type ._llcrypt unregistered [16059.904090] Key type ._llcrypt registered [16059.917836] Key type .llcrypt registered [16060.137590] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [16061.691716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16061.769264] alg: No test for adler32 (adler32-zlib) [16063.234720] Lustre: Lustre: Build Version: 2.17.57_117_g424a755 [16063.788814] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [16065.656323] Key type lgssc registered [16067.394763] Lustre: Echo OBD driver; http://www.lustre.org/ [16119.603341] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [16129.975254] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [16172.533835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16172.549789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16173.835250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16173.888664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16174.011869] Lustre: lustre-MDT0000: new disk, initializing [16174.148080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16174.186757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16178.957571] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [16193.081270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16193.349870] Lustre: 523124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16193.435413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16193.439292] Lustre: Skipped 1 previous similar message [16193.570725] Lustre: lustre-MDT0001: new disk, initializing [16193.740877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16193.810591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16193.819927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16198.921768] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [16209.715459] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16216.970394] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16225.598214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16225.850460] Lustre: lustre-OST0000: new disk, initializing [16225.853642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16225.860970] Lustre: 525093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16225.951996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16233.393281] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [16234.670751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16234.692489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16234.818121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16248.444970] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16252.879746] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16253.116805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16257.558771] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16257.849110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16260.581059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16260.581148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16260.590840] LustreError: Skipped 1 previous similar message [16260.625586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16260.636259] Lustre: Skipped 1 previous similar message [16265.803454] Lustre: server umount lustre-OST0000 complete [16270.822871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16270.823257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16270.829434] Lustre: Skipped 1 previous similar message [16270.830297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16275.938378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16275.948559] Lustre: Skipped 2 previous similar messages [16276.561221] Lustre: server umount lustre-MDT0000 complete [16281.304615] LustreError: 523115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854684 with bad export cookie 11556609282127688962 [16281.316747] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16281.327485] LustreError: 523115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16281.892622] Lustre: server umount lustre-MDT0001 complete [16311.426780] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [16322.468465] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [16368.107326] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16114 sec ========= 04:06:07 (1788854767) [16370.928697] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:06:10 (1788854770) === [16376.590693] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:06:16 (1788854776) === [16431.797630] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [16435.725507] Key type lgssc unregistered [16436.194298] LNet: 531745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16436.214769] LNetError: 531745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16436.255372] LNet: Removed LNI 192.168.204.115@tcp [16437.714063] Key type .llcrypt unregistered [16437.722772] Key type ._llcrypt unregistered