[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 533362355 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002348] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009120] LSM: Security Framework initializing [ 0.010070] Yama: becoming mindful. [ 0.012019] SELinux: Initializing. [ 0.013020] *** VALIDATE selinux *** [ 0.020305] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024709] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028098] *** VALIDATE tmpfs *** [ 0.030303] *** VALIDATE proc *** [ 0.031208] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033236] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.039865] debug: unmapping init [mem 0xffffffff93659000-0xffffffff93660fff] [ 0.041843] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042619] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043018] ... version: 2 [ 0.044009] ... bit width: 48 [ 0.045009] ... generic registers: 4 [ 0.046009] ... value mask: 0000ffffffffffff [ 0.047010] ... max period: 00007fffffffffff [ 0.048009] ... fixed-purpose events: 3 [ 0.049009] ... event mask: 000000070000000f [ 0.050262] rcu: Hierarchical SRCU implementation. [ 0.052357] smp: Bringing up secondary CPUs ... [ 0.053502] x86: Booting SMP configuration: [ 0.054019] .... node #0, CPUs: #1 #2 #3 [ 0.059100] smp: Brought up 1 node, 4 CPUs [ 0.061019] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.168000] node 0 deferred pages initialised in 101ms [ 0.172011] devtmpfs: initialized [ 0.173233] x86/mm: Memory block size: 128MB [ 0.175755] gcov: version magic: 0x41383552 [ 0.177134] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.178078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.179329] pinctrl core: initialized pinctrl subsystem [ 0.180261] [ 0.180780] ************************************************************* [ 0.181012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182012] ** ** [ 0.183017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.184012] ** ** [ 0.185017] ** This means that this kernel is built to expose internal ** [ 0.186014] ** IOMMU data structures, which may compromise security on ** [ 0.187013] ** your system. ** [ 0.188015] ** ** [ 0.189013] ** If you see this message and you are not debugging the ** [ 0.190016] ** kernel, report this immediately to your vendor! ** [ 0.191015] ** ** [ 0.192014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193012] ************************************************************* [ 0.194771] NET: Registered protocol family 16 [ 0.197683] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201187] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209013] cpuidle: using governor menu [ 0.210539] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215802] PCI: Using configuration type 1 for base access [ 0.218215] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.231164] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.234077] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.238160] cryptd: max_cpu_qlen set to 1000 [ 0.242275] ACPI: Added _OSI(Module Device) [ 0.245023] ACPI: Added _OSI(Processor Device) [ 0.246015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.253203] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.260389] ACPI: Interpreter enabled [ 0.263088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264014] ACPI: Using IOAPIC for interrupt routing [ 0.268144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.272415] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.286818] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.289659] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.293024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.298101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.305511] acpiphp: Slot [2] registered [ 0.307158] acpiphp: Slot [5] registered [ 0.309165] acpiphp: Slot [6] registered [ 0.311242] acpiphp: Slot [7] registered [ 0.312131] acpiphp: Slot [8] registered [ 0.314152] acpiphp: Slot [9] registered [ 0.316198] acpiphp: Slot [10] registered [ 0.318109] acpiphp: Slot [3] registered [ 0.320130] acpiphp: Slot [4] registered [ 0.321129] acpiphp: Slot [11] registered [ 0.322090] acpiphp: Slot [12] registered [ 0.325131] acpiphp: Slot [13] registered [ 0.327124] acpiphp: Slot [14] registered [ 0.328138] acpiphp: Slot [15] registered [ 0.330143] acpiphp: Slot [16] registered [ 0.332088] acpiphp: Slot [17] registered [ 0.333098] acpiphp: Slot [18] registered [ 0.334090] acpiphp: Slot [19] registered [ 0.336077] acpiphp: Slot [20] registered [ 0.337082] acpiphp: Slot [21] registered [ 0.339022] acpiphp: Slot [22] registered [ 0.340088] acpiphp: Slot [23] registered [ 0.341143] acpiphp: Slot [24] registered [ 0.343247] acpiphp: Slot [25] registered [ 0.345095] acpiphp: Slot [26] registered [ 0.347114] acpiphp: Slot [27] registered [ 0.349102] acpiphp: Slot [28] registered [ 0.351109] acpiphp: Slot [29] registered [ 0.352115] acpiphp: Slot [30] registered [ 0.353079] acpiphp: Slot [31] registered [ 0.354066] PCI host bridge to bus 0000:00 [ 0.356019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.358024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.360046] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.363027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.368057] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.370164] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.374198] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.378593] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.396016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.401065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.405028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.412037] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.416038] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.419419] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.421964] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.425044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.429100] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.434017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.447018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.454016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.456000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.464016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.471017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.495000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.503013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.528017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.539845] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.551019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.558020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.583023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.597696] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.608028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.614018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.634016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.648478] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.657020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.663053] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.682021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.694942] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.704018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.714020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.731019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.748681] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.752433] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.754544] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.756403] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.758285] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.763131] iommu: Default domain type: Passthrough [ 0.765562] SCSI subsystem initialized [ 0.767155] ACPI: bus type USB registered [ 0.768348] usbcore: registered new interface driver usbfs [ 0.770086] usbcore: registered new interface driver hub [ 0.772075] usbcore: registered new device driver usb [ 0.773207] pps_core: LinuxPPS API ver. 1 registered [ 0.775011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.776059] PTP clock support registered [ 0.779048] EDAC MC: Ver: 3.0.0 [ 0.781135] PCI: Using ACPI for IRQ routing [ 0.782974] NetLabel: Initializing [ 0.784015] NetLabel: domain hash size = 128 [ 0.786012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.789105] NetLabel: unlabeled traffic allowed by default [ 0.793157] vgaarb: loaded [ 0.795385] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.798022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.810705] clocksource: Switched to clocksource kvm-clock [ 0.947444] VFS: Disk quotas dquot_6.6.0 [ 0.949609] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.954799] *** VALIDATE ramfs *** [ 0.956720] *** VALIDATE hugetlbfs *** [ 0.959322] pnp: PnP ACPI init [ 0.962842] pnp: PnP ACPI: found 6 devices [ 0.980745] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.984203] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.987053] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.989215] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.991965] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.995113] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.998068] NET: Registered protocol family 2 [ 1.000669] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.006660] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.011696] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.019082] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.023401] TCP: Hash tables configured (established 65536 bind 65536) [ 1.027774] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.032577] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.036372] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.040479] NET: Registered protocol family 1 [ 1.043741] RPC: Registered named UNIX socket transport module. [ 1.048214] RPC: Registered udp transport module. [ 1.050256] RPC: Registered tcp transport module. [ 1.052465] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.056364] NET: Registered protocol family 44 [ 1.058132] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.061276] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.063228] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.064896] PCI: CLS 0 bytes, default 64 [ 1.067151] Unpacking initramfs... [ 2.589314] debug: unmapping init [mem 0xffff90c8fcc54000-0xffff90c8fffbffff] [ 2.593730] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.595683] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.598191] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.148644] Initialise system trusted keyrings [ 3.150908] Key type blacklist registered [ 3.155306] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.173037] zbud: loaded [ 3.177567] *** VALIDATE nfs *** [ 3.179170] *** VALIDATE nfs4 *** [ 3.182466] pstore: using deflate compression [ 3.187308] Platform Keyring initialized [ 3.295775] NET: Registered protocol family 38 [ 3.297749] Key type asymmetric registered [ 3.301454] Asymmetric key parser 'x509' registered [ 3.303497] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.306895] io scheduler mq-deadline registered [ 3.308877] io scheduler kyber registered [ 3.310594] io scheduler bfq registered [ 3.312706] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.317113] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.320421] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.323682] ACPI: Power Button [PWRF] [ 3.331432] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.341871] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.363693] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.372660] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.389132] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.418840] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.447141] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.451902] Non-volatile memory driver v1.3 [ 3.453733] Linux agpgart interface v0.103 [ 3.484766] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.488021] vda: detected capacity change from 0 to 76775424 [ 3.505709] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.510888] vdb: detected capacity change from 0 to 1073741824 [ 3.534721] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.538420] vdc: detected capacity change from 0 to 2621440000 [ 3.556398] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.559601] vdd: detected capacity change from 0 to 2621440000 [ 3.574956] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.578755] vde: detected capacity change from 0 to 4294967296 [ 3.593546] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.596641] vdf: detected capacity change from 0 to 4294967296 [ 3.605351] libphy: Fixed MDIO Bus: probed [ 3.632964] usbcore: registered new interface driver usbserial_generic [ 3.636069] usbserial: USB Serial support registered for generic [ 3.638973] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.645155] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.647703] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.650560] mousedev: PS/2 mouse device common for all mice [ 3.653677] rtc_cmos 00:05: RTC can wake from S4 [ 3.655683] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.658116] rtc_cmos 00:05: registered as rtc0 [ 3.662817] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.667103] intel_pstate: CPU model not supported [ 3.670273] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.677093] hid: raw HID events driver (C) Jiri Kosina [ 3.680488] usbcore: registered new interface driver usbhid [ 3.682574] usbhid: USB HID core driver [ 3.684043] drop_monitor: Initializing network drop monitor service [ 3.684668] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.685815] Initializing XFRM netlink socket [ 3.690586] NET: Registered protocol family 10 [ 3.693994] Segment Routing with IPv6 [ 3.695765] NET: Registered protocol family 17 [ 3.697702] mpls_gso: MPLS GSO support [ 3.704347] RAS: Correctable Errors collector initialized. [ 3.706422] AVX version of gcm_enc/dec engaged. [ 3.708183] AES CTR mode by8 optimization enabled [ 3.802715] sched_clock: Marking stable (3802629683, 0)->(4716487673, -913857990) [ 3.805990] registered taskstats version 1 [ 3.810524] Loading compiled-in X.509 certificates [ 3.812427] zswap: loaded using pool lzo/zbud [ 3.837475] Key type big_key registered [ 3.851235] Key type encrypted registered [ 3.852782] ima: No TPM chip found, activating TPM-bypass! [ 3.855445] ima: Allocated hash algorithm: sha1 [ 3.857563] ima: No architecture policies found [ 3.859610] evm: Initialising EVM extended attributes: [ 3.862197] evm: security.selinux [ 3.863583] evm: security.ima [ 3.865127] evm: security.capability [ 3.866755] evm: HMAC attrs: 0x1 [ 3.869481] rtc_cmos 00:05: setting system clock to 2026-09-12 02:52:12 UTC (1789181532) [ 3.877359] debug: unmapping init [mem 0xffffffff94603000-0xffffffff947fffff] [ 3.881041] debug: unmapping init [mem 0xffffffff93382000-0xffffffff93658fff] [ 3.892140] Write protecting the kernel read-only data: 28672k [ 3.896219] debug: unmapping init [mem 0xffffffff91a03000-0xffffffff91bfffff] [ 3.899444] debug: unmapping init [mem 0xffffffff92314000-0xffffffff923fffff] [ 3.960499] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.968000] systemd[1]: Detected virtualization kvm. [ 3.969619] systemd[1]: Detected architecture x86-64. [ 3.971309] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.002614] systemd[1]: No hostname configured. [ 4.004926] systemd[1]: Set hostname to . [ 4.007218] random: systemd: uninitialized urandom read (16 bytes read) [ 4.010371] systemd[1]: Initializing machine ID from random generator. [ 4.171818] random: systemd: uninitialized urandom read (16 bytes read) [ 4.174712] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.184616] random: systemd: uninitialized urandom read (16 bytes read) [ 4.187519] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.195944] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.897926] device-mapper: uevent: version 1.0.3 [ 4.900437] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.651826] virtio_net virtio0 ens2: renamed from eth0 [ 5.704219] random: fast init done [ 5.750212] scsi host0: ata_piix [ 5.803929] scsi host1: ata_piix [ 5.807493] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.810268] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.523670] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.372400] random: crng init done [ 10.373771] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.968293] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.280946] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.585941] SELinux: Disabled at runtime. [ 12.655213] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.662188] systemd[1]: Detected virtualization kvm. [ 12.664569] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.270097] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.273748] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.278920] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.283253] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.286493] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.300463] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.307290] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Startin[ 13.484388] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.876698] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.275749] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.329525] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.511228] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.534202] EDAC sbridge: Ver: 1.1.2 [ 16.991563] Key type dns_resolver registered [ 17.378476] NFS: Registering the id_resolver key type [ 17.380663] Key type id_resolver registered [ 17.382259] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg615-server login: [ 38.625010] hrtimer: interrupt took 6180851 ns [ 79.128914] libcfs: loading out-of-tree module taints kernel. [ 79.173191] Key type ._llcrypt registered [ 79.174848] Key type .llcrypt registered [ 79.246866] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 94.247255] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 95.361284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.382035] alg: No test for adler32 (adler32-zlib) [ 96.757175] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 97.483985] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 99.242039] Key type lgssc registered [ 100.620582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.092946] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 146.380347] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 157.273734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.318914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.581763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.611859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.728672] Lustre: lustre-MDT0000: new disk, initializing [ 158.836221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.865491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 163.300454] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.567829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.655177] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 174.691767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 174.696531] Lustre: Skipped 1 previous similar message [ 174.790621] Lustre: lustre-MDT0001: new disk, initializing [ 174.865283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.895148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.908304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.291499] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.164192] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.143385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.339671] Lustre: lustre-OST0000: new disk, initializing [ 189.343094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.348909] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 189.408868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.084765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.094886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.150183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 193.851780] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.779710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.933674] Lustre: lustre-OST0001: new disk, initializing [ 205.937212] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.949324] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.014247] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.018432] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.003887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.015738] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.107331] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 223.670333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.771629] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.713437] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.106562] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing yml_node [ 241.194825] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 243.390770] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 245.549474] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 246.770977] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 22:56:13 EDT 2026 [ 260.432554] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 261.496495] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 277.479953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 277.487580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.504112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 279.522303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.525990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 279.533799] Lustre: Skipped 1 previous similar message [ 279.537602] Lustre: Skipped 1 previous similar message [ 284.640554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.646867] Lustre: Skipped 2 previous similar messages [ 289.762139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.766038] Lustre: Skipped 3 previous similar messages [ 290.271233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 290.589331] Lustre: server umount lustre-MDT0000 complete [ 293.247703] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789181821 with bad export cookie 10421919044993061348 [ 293.251293] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.253719] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 294.880199] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.886782] Lustre: Skipped 1 previous similar message [ 294.890350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.578572] Lustre: server umount lustre-MDT0001 complete [ 303.494751] Lustre: server umount lustre-OST0000 complete [ 306.610230] Lustre: server umount lustre-OST0001 complete [ 311.808157] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 318.650727] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 346.408624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.573270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.592082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.651334] Lustre: lustre-MDT0000: new disk, initializing [ 346.707688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.722207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 349.734587] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 358.406818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.511926] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 358.563921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 358.570826] Lustre: Skipped 1 previous similar message [ 358.668524] Lustre: lustre-MDT0001: new disk, initializing [ 358.776922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 358.807559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 358.815322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 362.287804] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 370.244448] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 374.950553] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 380.608589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 380.847744] Lustre: lustre-OST0000: new disk, initializing [ 380.850515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 380.855548] Lustre: 17035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 380.926552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.427134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 382.434035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 382.523452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 385.400479] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 392.365624] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 396.152140] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 396.372667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 400.022496] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 400.228834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 402.915442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 402.918228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 402.921297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 402.921304] Lustre: Skipped 3 previous similar messages [ 402.921625] LustreError: Skipped 1 previous similar message [ 402.934563] Lustre: Skipped 2 previous similar messages [ 416.223151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 416.383264] Lustre: server umount lustre-OST0000 complete [ 420.327247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 420.337595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.358813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 420.367907] Lustre: Skipped 5 previous similar messages [ 425.679322] Lustre: server umount lustre-MDT0000 complete [ 428.512406] LustreError: 15967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.527283] LustreError: 15967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 428.814602] LustreError: 18376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789181957 with bad export cookie 10421919044993062881 [ 428.815440] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 428.831939] LustreError: 18376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 429.230264] Lustre: server umount lustre-MDT0001 complete [ 433.247106] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:59:19 (1789181959) [ 435.555871] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 437.030937] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:59:23 (1789181963) [ 442.571123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.962240] LustreError: 19449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 443.027219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.403138] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 448.482193] LustreError: 19450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.493940] LustreError: 19450:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 452.583786] LustreError: 19449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.801994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.237964] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 461.865724] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 465.859717] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 471.579994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.254087] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 483.775537] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 498.657943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.667209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.674080] LustreError: Skipped 1 previous similar message [ 498.689955] Lustre: Skipped 1 previous similar message [ 498.697127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.715854] Lustre: Skipped 2 previous similar messages [ 501.196846] Lustre: server umount lustre-OST0000 complete [ 504.294313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.537854] Lustre: server umount lustre-MDT0000 complete [ 513.557249] LustreError: 21289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789182042 with bad export cookie 10421919044993063763 [ 513.560878] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.572866] LustreError: 21289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 513.809103] Lustre: server umount lustre-MDT0001 complete [ 521.790984] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 524.097281] Key type lgssc unregistered [ 524.369427] LNet: 22974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 524.378139] LNetError: 22974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 524.395923] LNet: Removed LNI 192.168.206.115@tcp [ 525.223477] Key type .llcrypt unregistered [ 525.225505] Key type ._llcrypt unregistered [ 548.361651] Key type ._llcrypt registered [ 548.364457] Key type .llcrypt registered [ 548.528657] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 549.437374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 549.486899] alg: No test for adler32 (adler32-zlib) [ 550.557119] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 550.789106] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 552.415214] Key type lgssc registered [ 553.405806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 560.587319] loop: module loaded [ 560.636683] loop0: detected capacity change from 0 to 204800000 [ 560.856300] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 560.878404] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 561.275889] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 566.058837] loop0: detected capacity change from 0 to 204800000 [ 566.196199] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 566.205621] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 566.218299] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 570.445440] loop0: detected capacity change from 0 to 204800000 [ 575.346904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 575.376638] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.653426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 576.714684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 576.777255] Lustre: lustre-MDT0000: new disk, initializing [ 576.896479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.926499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 581.350982] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 589.142489] loop1: detected capacity change from 0 to 204800000 [ 594.338928] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.727775] Lustre: lustre-OST07c6: new disk, initializing [ 594.731873] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 594.735750] Lustre: Skipped 1 previous similar message [ 594.740255] Lustre: 26170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 594.813772] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 599.477819] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 604.196317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 604.206558] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 604.234245] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 609.322336] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 609.328863] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 609.334433] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 609.341199] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 609.349937] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 609.361571] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 630.320393] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 630.326601] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 630.332125] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 630.336349] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 630.340786] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 630.344692] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 634.732454] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 634.740234] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 634.746936] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 634.752716] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 634.760516] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 634.766608] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 634.771239] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 634.777297] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 634.782435] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 634.786671] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 634.793552] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 634.800787] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 637.963173] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 637.975234] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 637.982360] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 637.990664] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 637.996799] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 638.011511] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 638.020158] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 638.030637] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 638.044197] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 638.052224] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 638.061310] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 638.076364] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.795184] Lustre: 26178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 642.823202] Lustre: 26178:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 642.846527] Lustre: 26178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 642.856936] Lustre: 26178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.862182] Lustre: 26178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 642.869292] Lustre: 26178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.875981] Lustre: 26178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 642.882331] Lustre: 26178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.890384] Lustre: 26178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 642.896129] Lustre: 26178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.900948] Lustre: 26178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 642.906440] Lustre: 26178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.645634] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 652.656812] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 652.667366] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 652.679976] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 652.687602] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 652.701500] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 652.711678] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 652.717987] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 652.722486] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 652.726706] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 652.733470] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 652.740427] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 701.411430] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.433895] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 701.434800] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 706.528258] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 706.845769] Lustre: server umount lustre-OST07c6 complete [ 714.645865] Lustre: server umount lustre-MDT0000 complete [ 727.083372] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 729.453559] Key type lgssc unregistered [ 729.739866] LNet: 28725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.752332] LNetError: 28725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.768238] LNet: Removed LNI 192.168.206.115@tcp [ 730.531763] Key type .llcrypt unregistered [ 730.535316] Key type ._llcrypt unregistered [ 746.423341] Key type ._llcrypt registered [ 746.425151] Key type .llcrypt registered [ 746.551505] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 747.410560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 747.496067] alg: No test for adler32 (adler32-zlib) [ 748.519577] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 748.724112] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 750.351212] Key type lgssc registered [ 750.980585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 757.246750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 757.264520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.910706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.251236] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 768.480838] LustreError: 29972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.500857] LustreError: 29972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 772.766865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.345195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 778.444107] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 784.905486] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 789.174792] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 794.496500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 794.755553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 799.600905] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 803.948158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 805.843695] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 818.656664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 818.666973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.686217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.692394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.693357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.709133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 823.963297] Lustre: server umount lustre-OST0000 complete [ 829.922834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 829.927495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 829.928486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 829.928492] Lustre: Skipped 1 previous similar message [ 829.940819] Lustre: Skipped 1 previous similar message [ 833.272462] Lustre: server umount lustre-MDT0000 complete [ 835.042895] LustreError: 29972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.058888] LustreError: 29972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 836.303303] LustreError: 29952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789182364 with bad export cookie 1413678507571052218 [ 836.315443] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 836.317148] LustreError: 29952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 836.618734] Lustre: server umount lustre-MDT0001 complete [ 845.741147] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 848.801186] Key type lgssc unregistered [ 849.218118] LNet: 33703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.223070] LNetError: 33703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.255362] LNet: Removed LNI 192.168.206.115@tcp [ 850.457237] Key type .llcrypt unregistered [ 850.459112] Key type ._llcrypt unregistered [ 869.692995] Key type ._llcrypt registered [ 869.695779] Key type .llcrypt registered [ 869.837600] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 870.899325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.047978] alg: No test for adler32 (adler32-zlib) [ 872.125871] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 872.292896] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 874.040083] Key type lgssc registered [ 875.268383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 883.260596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 883.285517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.870570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.242447] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 894.438919] LustreError: 34970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.457965] LustreError: 34970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 898.327833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.784109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 904.027718] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 912.333533] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 917.560852] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 925.568646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 925.900984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.059040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 932.937334] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 940.401317] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 954.852642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 954.853578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.867641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 954.888913] Lustre: Skipped 1 previous similar message [ 959.982633] LustreError: 36795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.999734] LustreError: 36795:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 960.118827] Lustre: server umount lustre-OST0000 complete [ 965.097273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.102560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 965.118713] Lustre: Skipped 1 previous similar message [ 965.126828] Lustre: Skipped 2 previous similar messages [ 969.681947] Lustre: server umount lustre-MDT0000 complete [ 970.210178] LustreError: 34971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.219158] LustreError: 34971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 972.537191] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789182501 with bad export cookie 18176858824991686114 [ 972.543386] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.548660] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 972.757366] Lustre: server umount lustre-MDT0001 complete [ 981.050329] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 983.482889] Key type lgssc unregistered [ 983.723956] LNet: 38530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.731769] LNetError: 38530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.750761] LNet: Removed LNI 192.168.206.115@tcp [ 984.557410] Key type .llcrypt unregistered [ 984.559545] Key type ._llcrypt unregistered [ 1003.679473] Key type ._llcrypt registered [ 1003.681581] Key type .llcrypt registered [ 1003.752890] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1004.592636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.629828] alg: No test for adler32 (adler32-zlib) [ 1005.663434] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 1005.964877] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1007.711264] Key type lgssc registered [ 1009.048797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1016.372902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1016.398803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.767068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.135824] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1028.082322] LustreError: 39922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.102804] LustreError: 39922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1028.243930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.568813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1032.927301] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1039.944247] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1045.725404] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1051.648493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1052.054753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.305670] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1063.302898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1063.843809] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1069.543472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1069.552916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.578660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1070.403690] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 1071.311773] Lustre: server umount lustre-MDT0000 complete [ 1073.635812] LustreError: 39921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.638384] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.655164] LustreError: 39921:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1073.700565] Lustre: Skipped 1 previous similar message [ 1074.809171] LustreError: 39904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789182603 with bad export cookie 8154252600594803366 [ 1074.812476] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1074.829530] LustreError: 39904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1075.506794] LustreError: 39916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.528485] LustreError: 39916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1075.550688] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 1078.752702] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.753444] LustreError: 39918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.767865] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1081.186441] Lustre: server umount lustre-MDT0001 complete [ 1101.743990] Lustre: server umount lustre-OST0000 complete [ 1117.404884] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1120.345888] Key type lgssc unregistered [ 1120.637378] LNet: 43781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.648311] LNetError: 43781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1120.677745] LNet: Removed LNI 192.168.206.115@tcp [ 1121.540799] Key type .llcrypt unregistered [ 1121.542600] Key type ._llcrypt unregistered [ 1142.054771] Key type ._llcrypt registered [ 1142.057409] Key type .llcrypt registered [ 1142.145478] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1143.137144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1143.162661] alg: No test for adler32 (adler32-zlib) [ 1144.329416] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 1144.572201] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1146.263213] Key type lgssc registered [ 1147.208795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1153.919809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1153.931705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.344517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.441527] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1164.771068] LustreError: 45174:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.787718] LustreError: 45174:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1166.226919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.538804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1169.702361] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1175.355404] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1180.770213] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1187.441158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1187.939693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.138338] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1200.319407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1205.682329] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1212.737563] Lustre: lustre-OST0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 1212.897794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1212.901590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.971030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.971641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.971805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.974304] Lustre: Skipped 2 previous similar messages [ 1217.829576] LustreError: 46995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.948167] Lustre: server umount lustre-OST0000 complete [ 1236.449185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.464056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.470279] Lustre: Skipped 1 previous similar message [ 1240.554575] LustreError: 45902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.577940] LustreError: 45902:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1240.719116] Lustre: server umount lustre-MDT0000 complete [ 1243.960754] LustreError: 47868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789182772 with bad export cookie 3336435788002037564 [ 1243.963109] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.974394] LustreError: 47868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1250.322989] Lustre: server umount lustre-MDT0001 complete [ 1261.841364] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1264.686797] Key type lgssc unregistered [ 1264.940122] LNet: 48879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.948569] LNetError: 48879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.022636] LNet: Removed LNI 192.168.206.115@tcp [ 1266.925293] Key type .llcrypt unregistered [ 1266.927546] Key type ._llcrypt unregistered [ 1285.383713] Key type ._llcrypt registered [ 1285.386386] Key type .llcrypt registered [ 1285.463339] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1286.133363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1286.187103] alg: No test for adler32 (adler32-zlib) [ 1287.164158] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 1287.396458] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1289.063253] Key type lgssc registered [ 1290.018431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1296.975161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1296.996206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.417333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.556365] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1308.646710] LustreError: 50274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.668398] LustreError: 50274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1310.781770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.223416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1315.960578] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1323.321790] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1329.159602] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1335.120156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1335.482620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.488899] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1343.665572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1348.059985] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1353.758687] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1356.618914] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1369.341958] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1123h14m54s [ 1371.319897] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1374.690636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.699722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.708625] Lustre: Skipped 2 previous similar messages [ 1374.723647] Lustre: Skipped 2 previous similar messages [ 1375.785257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1377.634839] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 1377.642397] Lustre: Skipped 3 previous similar messages [ 1379.908677] Lustre: server umount lustre-MDT0000 complete [ 1381.859474] LustreError: 50275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.889180] LustreError: 50275:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1383.080507] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1388.007475] LustreError: 50274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.024667] LustreError: 50274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1388.106509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.263782] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1388.566531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.632254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1389.621712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1392.306886] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1395.139393] Lustre: DEBUG MARKER: Wait for df (36708) ... [ 1396.218749] Lustre: DEBUG MARKER: done [ 1403.372931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1403.382212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.422767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.890326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1406.218221] Lustre: server umount lustre-OST0000 complete [ 1410.017462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1410.021034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.033264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.043015] Lustre: Skipped 2 previous similar messages [ 1410.057226] Lustre: Skipped 4 previous similar messages [ 1415.649063] Lustre: server umount lustre-MDT0000 complete [ 1416.162395] LustreError: 52114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1416.210477] LustreError: 52114:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1419.261421] LustreError: 52903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789182947 with bad export cookie 7019017662386547819 [ 1419.270025] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1419.277592] LustreError: 52903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1419.623408] Lustre: server umount lustre-MDT0001 complete [ 1428.459969] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1430.981199] Key type lgssc unregistered [ 1431.249302] LNet: 55199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.261311] LNetError: 55199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.289581] LNet: Removed LNI 192.168.206.115@tcp [ 1431.929922] Key type .llcrypt unregistered [ 1431.934689] Key type ._llcrypt unregistered [ 1463.631054] Key type ._llcrypt registered [ 1463.634308] Key type .llcrypt registered [ 1463.702886] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1464.332374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.454247] alg: No test for adler32 (adler32-zlib) [ 1465.424953] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 1465.659023] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1467.319209] Key type lgssc registered [ 1468.146709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1474.363241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1474.391137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.602178] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1475.622826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1475.842084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.306321] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1486.642613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1486.751898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1486.804530] Lustre: 57381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1486.999559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.201752] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1495.450581] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1499.766496] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1506.043450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1506.243914] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1506.369613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1512.090415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1512.272846] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1520.456687] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1527.447583] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1123h17m33s [ 1528.628448] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1534.564533] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1536.798678] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1536.855195] LustreError: 59542:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1557.340802] Lustre: lustre-MDT0000: Client fe79b827-5d7c-4cbd-a7ab-ead59f194576 (at 192.168.206.15@tcp) reconnecting [ 1576.879118] LustreError: 59542:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1578.655310] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1581.184540] Lustre: DEBUG MARKER: done [ 1588.706831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1588.714215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.723274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1589.216594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1589.225431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1589.238770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1589.244096] Lustre: Skipped 1 previous similar message [ 1592.602891] Lustre: server umount lustre-OST0000 complete [ 1599.457939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.472514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.478070] Lustre: Skipped 1 previous similar message [ 1601.655457] Lustre: server umount lustre-MDT0000 complete [ 1604.853261] LustreError: 58124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789183133 with bad export cookie 17853529748643114728 [ 1604.858488] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1604.869103] LustreError: 58124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1605.177701] Lustre: server umount lustre-MDT0001 complete [ 1614.210956] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1616.557880] Key type lgssc unregistered [ 1616.879871] LNet: 61710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.893272] LNetError: 61710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.918730] LNet: Removed LNI 192.168.206.115@tcp [ 1617.666693] Key type .llcrypt unregistered [ 1617.668875] Key type ._llcrypt unregistered [ 1653.920469] Key type ._llcrypt registered [ 1653.925197] Key type .llcrypt registered [ 1654.057870] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1654.842971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1654.858633] alg: No test for adler32 (adler32-zlib) [ 1655.931092] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 1656.162681] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1657.815250] Key type lgssc registered [ 1659.154860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1663.084211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1663.108399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.443464] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1664.470456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1664.672617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.540942] Lustre: Failing over lustre-MDT0000 [ 1665.739167] Lustre: server umount lustre-MDT0000 complete [ 1672.466429] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:19:58 (1789183198) [ 1678.994015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.450959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.547766] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1690.211872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.277982] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1690.297735] Lustre: 64488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1690.398225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1690.467833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1693.176167] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1698.899601] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1703.538536] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1708.326878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1708.455156] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1708.618183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.807330] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1714.322359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1718.813387] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1724.300627] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1729.913949] Lustre: server umount lustre-MDT0000 complete [ 1730.027267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.052167] Lustre: Skipped 1 previous similar message [ 1733.040890] LustreError: 64482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789183261 with bad export cookie 1746439207822179741 [ 1733.047721] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.049578] LustreError: 64482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1733.296856] Lustre: server umount lustre-MDT0001 complete [ 1736.528692] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1740.525882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1747.770582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.498394] Lustre: 63408:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789183263/real 1789183263] req@0000000054d96598 x1876094546381952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789183279 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1750.525362] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1750.535323] Lustre: Skipped 1 previous similar message [ 1753.632254] Lustre: 63408:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789183266/real 1789183266] req@00000000d916a895 x1876094546382080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789183282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1755.616089] Lustre: 63410:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789183268/real 1789183268] req@0000000050380b81 x1876094546382208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789183284 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1757.738535] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x183c9934dc0e4680 [ 1757.748220] Lustre: MGC192.168.206.115@tcp: Connection restored to 0@lo (at 0@lo) [ 1758.157840] LustreError: 66336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1758.176364] LustreError: 66336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1758.263185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.379527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1759.270431] Lustre: 63411:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789183271/real 1789183271] req@00000000bbcd8661 x1876094546382464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789183287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1761.316312] LustreError: 68257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.338095] LustreError: 68257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1762.259225] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1765.347115] Lustre: 63409:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789183277/real 1789183277] req@00000000db3698c2 x1876094546382592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789183293 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1766.369608] LustreError: 68255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.405251] LustreError: 68255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1769.445509] LustreError: 68256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1769.462690] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1769.477450] LustreError: 68256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1771.706606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.123874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1775.989266] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1777.143581] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1783.031292] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1788.364060] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1797.601717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.610054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.619916] Lustre: Skipped 2 previous similar messages [ 1797.624022] Lustre: Skipped 1 previous similar message [ 1802.723642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.742178] Lustre: Skipped 3 previous similar messages [ 1807.855215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1807.861247] Lustre: Skipped 1 previous similar message [ 1809.375213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1809.628539] Lustre: server umount lustre-MDT0000 complete [ 1812.708687] LustreError: 64482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789183341 with bad export cookie 1746439207822182016 [ 1812.710701] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1812.715930] LustreError: 64482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1812.961138] Lustre: server umount lustre-MDT0001 complete [ 1818.442329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1826.831612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.343154] Lustre: 63409:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789183341/real 1789183341] req@0000000055053213 x1876094546411264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789183357 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1829.371025] Lustre: 63409:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1829.375322] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1837.599830] LustreError: 63407:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a19ecc14 x1876094546412032/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1837.854169] LustreError: 68992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.916166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.975471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1842.055722] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1848.288943] LustreError: 66337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.320704] LustreError: 66337:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1849.364866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.451807] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1854.964782] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1859.652062] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1863.155985] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1865.819366] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1873.391227] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.412074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.412745] Lustre: Skipped 1 previous similar message [ 1873.453496] Lustre: Skipped 2 previous similar messages [ 1883.617534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1883.623610] Lustre: Skipped 5 previous similar messages [ 1887.199211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1887.387534] Lustre: server umount lustre-MDT0000 complete [ 1888.736806] LustreError: 68992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1888.749893] LustreError: 68992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1890.673050] LustreError: 65238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789183419 with bad export cookie 1746439207822183395 [ 1890.675474] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1890.678740] LustreError: 65238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1890.933636] Lustre: server umount lustre-MDT0001 complete [ 1897.111657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1900.099629] Lustre: DEBUG MARKER: files compared the same [ 1912.027874] Lustre: server umount lustre-OST0000 complete [ 1927.319231] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1929.834069] Key type lgssc unregistered [ 1930.048175] LNet: 74214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.056472] LNetError: 74214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.088458] LNet: Removed LNI 192.168.206.115@tcp [ 1930.854549] Key type .llcrypt unregistered [ 1930.856118] Key type ._llcrypt unregistered [ 1949.710428] Key type ._llcrypt registered [ 1949.713767] Key type .llcrypt registered [ 1949.837866] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1950.636478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1950.788954] alg: No test for adler32 (adler32-zlib) [ 1951.876674] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 1952.064546] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1953.743670] Key type lgssc registered [ 1955.056500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1962.671457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1962.700884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.154878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.188541] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug +malloc all [ 1973.733463] LustreError: 75607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.759364] LustreError: 75607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1977.231878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.514386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1981.493781] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug +malloc all [ 1988.208742] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1993.143696] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1998.817251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1998.990561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.925977] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug +malloc all [ 2009.260267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2011.593496] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2018.785218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.790964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.802884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2019.808487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2019.808839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.812229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2024.920092] Lustre: server umount lustre-OST0000 complete [ 2029.025421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2029.032896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.053527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.056977] Lustre: Skipped 1 previous similar message [ 2035.171706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.179905] Lustre: Skipped 3 previous similar messages [ 2040.288072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2040.293232] Lustre: Skipped 1 previous similar message [ 2042.335164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2042.544026] Lustre: server umount lustre-MDT0000 complete [ 2044.385553] LustreError: 75603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.402569] LustreError: 75603:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2045.616825] LustreError: 75589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789183574 with bad export cookie 2833466540338729144 [ 2045.622966] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2045.628142] LustreError: 75589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2045.845738] Lustre: server umount lustre-MDT0001 complete [ 2054.818789] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2057.637609] Key type lgssc unregistered [ 2058.005475] LNet: 79139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.011258] LNetError: 79139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.047225] LNet: Removed LNI 192.168.206.115@tcp [ 2058.994556] Key type .llcrypt unregistered [ 2058.998753] Key type ._llcrypt unregistered [ 2080.488415] Key type ._llcrypt registered [ 2080.491325] Key type .llcrypt registered [ 2080.588439] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2081.787902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2081.820929] alg: No test for adler32 (adler32-zlib) [ 2082.853498] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 2083.086390] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2084.736347] Key type lgssc registered [ 2085.844181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2092.640347] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2092.666601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2108.191492] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2113.312638] LustreError: 80527:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.115@tcp: failed processing log, type 4: rc = -110 [ 2139.999187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.869258] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2152.571405] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2159.746744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.889735] Lustre: *** cfs_fail_loc=706, val=0*** [ 2159.898927] LustreError: 79861:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000eed9f7be x1876094994090240/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2160.474120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.606507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2164.824646] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2165.733758] LustreError: 80553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.748422] LustreError: 80553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2170.851043] LustreError: 80552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.659453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.177478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2178.117949] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2185.249814] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2190.409635] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2193.893133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2193.896496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.915915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2195.938546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2195.943508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.293838] Lustre: server umount lustre-OST0000 complete [ 2204.128212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2204.136867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.152370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.161258] Lustre: Skipped 1 previous similar message [ 2206.177430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.177837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.194411] Lustre: Skipped 1 previous similar message [ 2207.578460] Lustre: server umount lustre-MDT0000 complete [ 2210.562383] LustreError: 80535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789183739 with bad export cookie 13958878970081076658 [ 2210.571957] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2210.577986] LustreError: 80535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2210.849382] Lustre: server umount lustre-MDT0001 complete [ 2220.226406] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2223.233537] Key type lgssc unregistered [ 2223.448966] LNet: 84140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.456724] LNetError: 84140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.487108] LNet: Removed LNI 192.168.206.115@tcp [ 2225.125549] Key type .llcrypt unregistered [ 2225.131581] Key type ._llcrypt unregistered [ 2246.688678] Key type ._llcrypt registered [ 2246.691048] Key type .llcrypt registered [ 2246.851516] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2247.483477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2247.740887] alg: No test for adler32 (adler32-zlib) [ 2248.999712] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 2249.144972] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2250.855217] Key type lgssc registered [ 2252.118309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2259.605728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2259.618098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.645965] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2271.834493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.304636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2276.318931] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2277.349382] LustreError: 86241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.371839] LustreError: 86241:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2281.443318] LustreError: 86240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.461880] LustreError: 86240:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2283.983294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2284.256953] LustreError: 86241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.284718] LustreError: 86241:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2284.286441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.635489] LustreError: 86239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.658614] LustreError: 86239:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2289.733836] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2294.757352] LustreError: 86244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.792124] LustreError: 86244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2297.806415] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.818489] Lustre: MGS: Client 43acfec4-2b0d-4db7-8c46-714c68280b03 (at 0@lo) reconnecting [ 2297.857025] LustreError: 86255:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2298.095903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.178515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2302.907893] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2318.828629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.839059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2318.839928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2322.498737] Lustre: server umount lustre-OST0000 complete [ 2329.057721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.059245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.064627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2329.080460] Lustre: Skipped 2 previous similar messages [ 2329.110845] Lustre: Skipped 2 previous similar messages [ 2334.187673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.192709] Lustre: Skipped 1 previous similar message [ 2339.304581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2339.318256] Lustre: Skipped 1 previous similar message [ 2340.319123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2340.487948] Lustre: server umount lustre-MDT0000 complete [ 2344.350262] Lustre: server umount lustre-MDT0001 complete [ 2348.105230] Lustre: server umount MGS complete [ 2357.573897] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:31:24 (1789183884) [ 2379.759908] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 2387.734619] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2417.858211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.281755] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2432.205481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.387914] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2432.417990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2432.485511] Lustre: lustre-MDT0001: new disk, initializing [ 2432.553931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2432.607722] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2432.621146] Lustre: 93422:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2432.638071] LustreError: 93422:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2437.304588] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2446.398584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.632177] Lustre: lustre-OST0000: new disk, initializing [ 2446.635777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2446.643713] Lustre: 94187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2446.710469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.201265] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2456.609056] Lustre: 94679:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000049b64e73 x1876095168192512/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789184001 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2460.518557] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2466.859500] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.872803] Lustre: MGS: Client eaceb3b1-5d8c-4765-a335-29fc5c49f2c5 (at 0@lo) reconnecting [ 2466.910436] Lustre: 95233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2466.923419] LustreError: 93408:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2466.948520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2467.010725] Lustre: lustre-MDT0000: new disk, initializing [ 2467.143625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.170815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2467.282619] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2468.199953] Lustre: 95257:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a179b5f2 x1876095168199040/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789184012 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2471.802565] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2477.029184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2477.039383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2477.435670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2502.624374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2502.640132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.652119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.834368] Lustre: server umount lustre-OST0000 complete [ 2516.450630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.458995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2516.468258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2516.468268] Lustre: Skipped 3 previous similar messages [ 2516.473225] Lustre: Skipped 2 previous similar messages [ 2516.510553] LustreError: Skipped 1 previous similar message [ 2518.408818] Lustre: server umount lustre-MDT0000 complete [ 2521.593329] LustreError: 93393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.620895] LustreError: 93393:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2528.503956] Lustre: server umount lustre-MDT0001 complete [ 2532.422827] Lustre: server umount MGS complete [ 2542.996032] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:34:29 (1789184069) [ 2559.573699] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2562.152562] Key type lgssc unregistered [ 2562.494635] LNet: 97756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2562.500503] LNetError: 97756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2562.511983] LNet: Removed LNI 192.168.206.115@tcp [ 2563.287590] Key type .llcrypt unregistered [ 2563.289299] Key type ._llcrypt unregistered [ 2578.509778] Key type ._llcrypt registered [ 2578.514023] Key type .llcrypt registered [ 2578.639380] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2580.013727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2580.028619] alg: No test for adler32 (adler32-zlib) [ 2581.000251] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 2581.205890] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2582.881278] Key type lgssc registered [ 2583.892657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2590.149334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2590.177634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.551371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.208581] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2601.953399] LustreError: 98966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.966691] LustreError: 98966:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2602.156086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.386037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2605.868462] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2613.163363] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2619.118622] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2625.506366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2625.864872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.423838] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2637.164649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2639.377048] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2647.527064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.529654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2647.539432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2647.559071] Lustre: Skipped 1 previous similar message [ 2652.453506] Lustre: server umount lustre-OST0000 complete [ 2657.762546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.770395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.775333] Lustre: Skipped 1 previous similar message [ 2657.782454] Lustre: Skipped 2 previous similar messages [ 2661.836202] Lustre: server umount lustre-MDT0000 complete [ 2662.882495] LustreError: 98966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.913814] LustreError: 98966:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2664.987687] LustreError: 98946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789184193 with bad export cookie 13761743426040150478 [ 2664.990320] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2665.004277] LustreError: 98946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2665.283202] Lustre: server umount lustre-MDT0001 complete [ 2672.279476] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2674.494487] Key type lgssc unregistered [ 2674.772585] LNet: 102498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2674.778504] LNetError: 102498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2674.796870] LNet: Removed LNI 192.168.206.115@tcp [ 2675.554937] Key type .llcrypt unregistered [ 2675.557393] Key type ._llcrypt unregistered [ 2690.623598] Key type ._llcrypt registered [ 2690.628649] Key type .llcrypt registered [ 2690.765654] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2691.943390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2692.136684] alg: No test for adler32 (adler32-zlib) [ 2693.274837] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 2693.489145] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2695.159164] Key type lgssc registered [ 2696.344154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2702.367539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2702.372116] LustreError: 103572:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2702.397982] LustreError: 103579:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2702.404407] LustreError: 103572:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4973 [ 2702.418205] LustreError: 103572:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2702.422091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.428769] LustreError: 103572:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2704.146707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.238038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.625188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.931924] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2721.718179] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing lsmod [ 2724.196932] LustreError: 104783:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2724.198837] LustreError: 104784:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2724.217640] LustreError: 104783:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 2724.225170] LustreError: 104783:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2724.231657] LustreError: 104783:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2724.246336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2724.657636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.905383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2727.910798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.921777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.953577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.953910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.982222] Lustre: Skipped 1 previous similar message [ 2733.555962] Lustre: server umount lustre-MDT0000 complete [ 2735.073962] LustreError: 103616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.092917] LustreError: 103616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2736.853866] LustreError: 103593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789184265 with bad export cookie 10453725443237737802 [ 2736.866711] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2736.876745] Lustre: 103608:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2737.299771] Lustre: server umount lustre-MDT0001 complete [ 2751.662757] Lustre: server umount lustre-OST0000 complete [ 2759.598320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.153474] LustreError: 105848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.282847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2764.992805] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2765.282522] LustreError: 105849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.379981] LustreError: 105848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.975983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.447723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2777.604562] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2783.851596] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2789.186653] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2794.698850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2794.954421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.054487] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2804.250616] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2805.175146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2815.458374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.461144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2815.465748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2815.465756] Lustre: Skipped 2 previous similar messages [ 2815.471330] Lustre: Skipped 1 previous similar message [ 2820.576752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2820.582430] Lustre: Skipped 2 previous similar messages [ 2820.680558] Lustre: server umount lustre-OST0000 complete [ 2824.672571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2824.689100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.719423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2830.471592] Lustre: server umount lustre-MDT0000 complete [ 2830.817518] LustreError: 105849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.839467] LustreError: 105849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2833.254983] LustreError: 107691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789184361 with bad export cookie 10453725443237738474 [ 2833.262535] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2833.270525] LustreError: 107691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2833.559313] Lustre: server umount lustre-MDT0001 complete [ 2843.116873] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2845.790681] Key type lgssc unregistered [ 2846.077296] LNet: 109378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2846.089318] LNetError: 109378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2846.125250] LNet: Removed LNI 192.168.206.115@tcp [ 2847.085160] Key type .llcrypt unregistered [ 2847.089632] Key type ._llcrypt unregistered [ 2872.128639] Key type ._llcrypt registered [ 2872.135927] Key type .llcrypt registered [ 2872.229811] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2873.125605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2873.215296] alg: No test for adler32 (adler32-zlib) [ 2874.229780] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 2874.461078] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2876.103810] Key type lgssc registered [ 2877.209993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2884.789823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2884.811242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.351411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.976629] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2895.845071] LustreError: 110791:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.874842] LustreError: 110791:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2898.780796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.200094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2903.229637] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2909.607863] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2914.466870] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2920.970977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.350827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.801648] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2930.609814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2933.304883] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2938.930847] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2943.118652] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2949.089618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2949.110631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.120383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.139852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.140957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.144235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.020239] Lustre: server umount lustre-OST0000 complete [ 2960.864649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.875906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.902427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.912051] Lustre: Skipped 1 previous similar message [ 2964.288856] Lustre: server umount lustre-MDT0000 complete [ 2966.498825] LustreError: 110788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.515502] LustreError: 110788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2967.881798] LustreError: 113324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789184496 with bad export cookie 10726003702624076577 [ 2967.903580] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.914061] LustreError: 113324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2968.220750] Lustre: server umount lustre-MDT0001 complete [ 2977.544639] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2980.106611] Key type lgssc unregistered [ 2980.396213] LNet: 114428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2980.406756] LNetError: 114428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2981.484419] LNet: Removed LNI 192.168.206.115@tcp [ 2982.332563] Key type .llcrypt unregistered [ 2982.336274] Key type ._llcrypt unregistered [ 2997.006708] Key type ._llcrypt registered [ 2997.008753] Key type .llcrypt registered [ 2997.117907] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2998.098634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2998.227667] alg: No test for adler32 (adler32-zlib) [ 2999.260725] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 2999.524387] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3001.200530] Key type lgssc registered [ 3002.275676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3008.899454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3008.915473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.280696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.640106] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3019.746384] LustreError: 115645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.770994] LustreError: 115645:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3020.043393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.434297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.242968] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3030.916683] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3035.955032] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3042.159513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3042.462814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.503482] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3054.024560] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3054.783542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3067.879340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3067.884586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.890719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3070.433902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3070.436858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3070.438341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.441903] Lustre: Skipped 1 previous similar message [ 3073.093128] Lustre: server umount lustre-OST0000 complete [ 3077.091612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3077.098862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.113127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.674034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.677320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.667661] Lustre: server umount lustre-MDT0000 complete [ 3085.793926] LustreError: 116722:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.813688] LustreError: 116722:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3085.879756] LustreError: 115627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789184614 with bad export cookie 9420948249687864819 [ 3085.887133] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3085.896278] LustreError: 115627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3086.239972] Lustre: server umount lustre-MDT0001 complete [ 3094.938618] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3097.430376] Key type lgssc unregistered [ 3097.774551] LNet: 119196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3097.779324] LNetError: 119196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3097.800947] LNet: Removed LNI 192.168.206.115@tcp [ 3098.640834] Key type .llcrypt unregistered [ 3098.644515] Key type ._llcrypt unregistered [ 3121.439171] Key type ._llcrypt registered [ 3121.441333] Key type .llcrypt registered [ 3121.531527] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3122.478205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3122.567500] alg: No test for adler32 (adler32-zlib) [ 3123.619779] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 3123.895267] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3125.575230] Key type lgssc registered [ 3127.021140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3133.763986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3133.797644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.165280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.890283] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3144.677024] LustreError: 120609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3144.695907] LustreError: 120609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3145.013993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.408735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3149.149199] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3155.528811] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3160.391659] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3166.052904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3166.410908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.059663] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3177.119404] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3179.756569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3188.016597] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3190.936844] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3190.944047] Lustre: Skipped 2 previous similar messages [ 3195.022599] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3195.067115] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3195.453059] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3195.462866] Lustre: Skipped 2 previous similar messages [ 3202.127583] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3202.129851] Lustre: Skipped 1 previous similar message [ 3206.972565] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3207.000985] Lustre: Skipped 2 previous similar messages [ 3210.822530] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.15@tcp 192.168.0.[2,10]@tcp [ 3218.877384] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.15/24@tcp in log lustre-MDT0000 [ 3218.887363] Lustre: Skipped 2 previous similar messages [ 3219.666839] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.15/24@tcp [ 3219.673204] Lustre: Skipped 1 previous similar message [ 3231.205347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.210309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3231.213892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3231.225816] Lustre: Skipped 1 previous similar message [ 3231.259923] Lustre: Skipped 1 previous similar message [ 3233.264758] Lustre: server umount lustre-OST0000 complete [ 3237.859773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3237.869641] LustreError: Skipped 1 previous similar message [ 3237.874238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.890807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.440509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.442464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.560589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.569496] Lustre: Skipped 2 previous similar messages [ 3250.659368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3250.668800] Lustre: Skipped 1 previous similar message [ 3251.167198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3251.366397] Lustre: server umount lustre-MDT0000 complete [ 3254.806764] LustreError: 120591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789184783 with bad export cookie 12910799803371601208 [ 3254.815046] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3254.820041] LustreError: 120591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3255.161663] Lustre: server umount lustre-MDT0001 complete [ 3264.922526] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3267.647561] Key type lgssc unregistered [ 3267.958064] LNet: 125409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3267.981088] LNetError: 125409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3268.012363] LNet: Removed LNI 192.168.206.115@tcp [ 3268.821950] Key type .llcrypt unregistered [ 3268.829100] Key type ._llcrypt unregistered [ 3293.255213] Key type ._llcrypt registered [ 3293.257382] Key type .llcrypt registered [ 3293.327930] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3294.240682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3294.270618] alg: No test for adler32 (adler32-zlib) [ 3295.330504] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 3295.565440] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3297.247200] Key type lgssc registered [ 3298.280069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3305.294878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3305.317121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.755709] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3306.769397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.780410] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.15/24@tcp [ 3311.090939] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3317.220473] LustreError: 126945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3317.238429] LustreError: 126945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3318.542474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.863372] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3318.890047] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.15/24@tcp [ 3318.954538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3323.416713] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3330.136696] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3334.703878] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3341.487763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3341.876543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.262779] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3352.123682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3355.786366] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3364.320676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3364.338304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.365078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.838042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3364.844916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.869532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3367.408139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3367.415830] Lustre: Skipped 1 previous similar message [ 3370.407445] Lustre: server umount lustre-OST0000 complete [ 3375.081141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3375.096943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.121589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.646156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.752677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.766721] Lustre: Skipped 3 previous similar messages [ 3388.383804] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3388.620432] Lustre: server umount lustre-MDT0000 complete [ 3391.746075] LustreError: 127685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789184920 with bad export cookie 12101814911562906702 [ 3391.751376] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3391.767935] LustreError: 127685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3392.067325] Lustre: server umount lustre-MDT0001 complete [ 3401.263471] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3403.773782] Key type lgssc unregistered [ 3404.099102] LNet: 130533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3404.103129] LNetError: 130533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3404.121994] LNet: Removed LNI 192.168.206.115@tcp [ 3404.806526] Key type .llcrypt unregistered [ 3404.809571] Key type ._llcrypt unregistered [ 3427.578695] Key type ._llcrypt registered [ 3427.581270] Key type .llcrypt registered [ 3427.699497] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 3440.447419] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3441.941153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3441.957797] alg: No test for adler32 (adler32-zlib) [ 3443.038775] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 3443.253390] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3444.919242] Key type lgssc registered [ 3446.500208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3480.996168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3481.019031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.347275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3482.408467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3482.653241] Lustre: lustre-MDT0000: new disk, initializing [ 3482.748341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.765104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3488.241883] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3499.265724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.398363] Lustre: 133911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3499.489996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3499.496385] Lustre: Skipped 1 previous similar message [ 3499.580049] Lustre: lustre-MDT0001: new disk, initializing [ 3499.689853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3499.754567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3499.767988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3503.997776] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3511.258593] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3516.472783] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3522.292706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3522.512209] Lustre: lustre-OST0000: new disk, initializing [ 3522.514931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3522.520248] Lustre: 135877:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3522.603566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3527.724604] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3529.259864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3529.272452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3529.373843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3535.894897] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3539.739903] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3539.965201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3544.634849] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3544.807464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3549.665153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3549.680754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3549.710037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3550.191161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.195697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3552.871809] Lustre: server umount lustre-OST0000 complete [ 3560.420305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.425968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3560.441851] Lustre: Skipped 1 previous similar message [ 3560.464173] Lustre: Skipped 3 previous similar messages [ 3562.680479] Lustre: server umount lustre-MDT0000 complete [ 3566.037709] LustreError: 133904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185094 with bad export cookie 15241022477574151887 [ 3566.043619] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3566.052661] LustreError: 133904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3566.392210] Lustre: server umount lustre-MDT0001 complete [ 3573.689514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.062381] LustreError: 138095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.078593] LustreError: 138095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3574.125943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3578.100213] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3579.362551] LustreError: 138096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3583.461684] LustreError: 138095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.281842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.665965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3590.480107] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3597.176569] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3601.932243] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3608.062680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3608.381246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.251547] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3620.552613] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3624.813349] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3625.034687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3629.915539] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3630.080627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3639.109043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3639.287555] Lustre: lustre-OST0001: new disk, initializing [ 3639.290174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3639.296209] Lustre: 141467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3639.370382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3645.179934] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3649.117565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3649.133052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3649.218425] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3652.990050] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3653.174691] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3657.469843] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3657.669208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3662.486547] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a4b079a0800.ost_server_uuid 50 [ 3663.697268] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a4b079a0800.ost_server_uuid in IDLE state after 0 sec [ 3675.111497] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3675.126347] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.127535] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3675.128433] LustreError: Skipped 1 previous similar message [ 3675.162445] Lustre: Skipped 1 previous similar message [ 3676.763225] Lustre: server umount lustre-OST0001 complete [ 3680.232151] LustreError: 139916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.253793] LustreError: 139916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3683.296801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3683.304525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.319251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.330951] Lustre: Skipped 1 previous similar message [ 3685.357070] LustreError: 139915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.380367] LustreError: 139915:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3686.431813] Lustre: server umount lustre-OST0000 complete [ 3690.466687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3690.475138] Lustre: Skipped 3 previous similar messages [ 3695.898144] Lustre: server umount lustre-MDT0000 complete [ 3698.939250] LustreError: 138075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185227 with bad export cookie 15241022477574152769 [ 3698.943296] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3698.949829] LustreError: 138075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3699.257121] Lustre: server umount lustre-MDT0001 complete [ 3714.813153] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3717.161164] Key type lgssc unregistered [ 3717.422685] LNet: 144464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3717.430250] LNetError: 144464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3717.452742] LNet: Removed LNI 192.168.206.115@tcp [ 3718.194911] Key type .llcrypt unregistered [ 3718.198106] Key type ._llcrypt unregistered [ 3751.002620] Key type ._llcrypt registered [ 3751.006262] Key type .llcrypt registered [ 3751.111369] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 3762.443742] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3763.285453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3763.306772] alg: No test for adler32 (adler32-zlib) [ 3764.294762] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 3764.481416] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3766.167218] Key type lgssc registered [ 3767.026778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3798.397676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3798.440032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.807723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3799.828257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3799.907798] Lustre: lustre-MDT0000: new disk, initializing [ 3799.967622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.982724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3803.762808] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3815.181717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.320269] Lustre: 148597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3815.480658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3815.488124] Lustre: Skipped 1 previous similar message [ 3815.586461] Lustre: lustre-MDT0001: new disk, initializing [ 3815.645493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3815.690700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3815.717625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3821.086942] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3829.836923] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3834.380234] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3840.103955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3840.356154] Lustre: lustre-OST0000: new disk, initializing [ 3840.360089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3840.368486] Lustre: 150566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3840.499502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.995873] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3846.240607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3846.261516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3846.353666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3853.886379] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3858.184358] Lustre: Failing over lustre-OST0000 [ 3858.307991] Lustre: server umount lustre-OST0000 complete [ 3874.930345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3875.227091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.247146] Lustre: Skipped 1 previous similar message [ 3875.259137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3875.276535] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3877.179571] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3877.352899] Lustre: 146821:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185389/real 1789185389] req@000000004ce17c3b x1876096757824896/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789185405 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3878.239848] Lustre: 146822:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185390/real 1789185390] req@00000000eb3faa57 x1876096757825920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789185406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3880.445909] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3880.447853] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 3880.532822] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3883.487123] Lustre: 146821:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185395/real 1789185395] req@0000000014c906c4 x1876096757827328/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789185411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3883.511537] Lustre: 146821:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 3883.698264] Lustre: Failing over lustre-MDT0000 [ 3883.879988] Lustre: server umount lustre-MDT0000 complete [ 3885.537030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.548420] LustreError: 148604:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3885.549009] Lustre: Skipped 1 previous similar message [ 3885.564638] LustreError: 148604:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3887.421112] LustreError: 148603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.647161] Lustre: 146821:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185400/real 1789185400] req@00000000bee95423 x1876096757828352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789185416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3890.657044] LustreError: 150063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.672973] LustreError: 150063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3895.779884] LustreError: 148604:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3895.796950] LustreError: 148604:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3900.383112] Lustre: 148600:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789185412/real 1789185412] req@00000000ca78d55d x1876096757836032/t0(0) o101->MGC192.168.206.115@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1789185428 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3900.406165] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3900.419957] LustreError: 148608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3900.420024] Lustre: 148600:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3900.433620] LustreError: 148608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3900.635492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.625464] LustreError: 148608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.629376] LustreError: 146818:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000918bc3b6 x1876096757841536/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3910.635624] LustreError: 148608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 3910.968823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.010249] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3912.505901] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3914.769794] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3916.263983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3916.269419] Lustre: Skipped 1 previous similar message [ 3916.282741] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3916.329430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3921.892541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3921.900755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.937947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3926.501873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3926.503453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3926.517710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3927.584756] Lustre: server umount lustre-OST0000 complete [ 3931.617173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.622161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3931.622413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.622417] Lustre: Skipped 1 previous similar message [ 3931.628703] Lustre: Skipped 1 previous similar message [ 3936.736852] LustreError: 148607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3936.749748] LustreError: 148607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3938.364586] Lustre: server umount lustre-MDT0000 complete [ 3941.002851] LustreError: 148590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185469 with bad export cookie 7546414744031736786 [ 3941.008258] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3941.015991] LustreError: 148590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3941.251104] Lustre: server umount lustre-MDT0001 complete [ 3949.189505] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3951.277397] Key type lgssc unregistered [ 3951.496794] LNet: 154220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3951.502861] LNetError: 154220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3951.521078] LNet: Removed LNI 192.168.206.115@tcp [ 3952.157258] Key type .llcrypt unregistered [ 3952.160065] Key type ._llcrypt unregistered [ 3984.452103] Key type ._llcrypt registered [ 3984.457759] Key type .llcrypt registered [ 3984.581094] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3985.935091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3985.958058] alg: No test for adler32 (adler32-zlib) [ 3987.084128] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 3987.329895] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3989.032134] Key type lgssc registered [ 3990.005852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3997.196846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3997.219241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3998.711102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.516315] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4008.928016] LustreError: 155750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4008.948965] LustreError: 155750:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4010.570405] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4013.680090] LustreError: 155778:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4013.694349] LustreError: 155778:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4013.707362] LustreError: 155778:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4014.035101] Lustre: server umount lustre-MDT0000 complete [ 4027.496277] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4036.346538] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4064.662637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.867238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4064.899672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4064.985519] Lustre: lustre-MDT0000: new disk, initializing [ 4065.091695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4065.109183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4068.567089] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4076.805402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.919563] Lustre: 159077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4076.991215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4076.996330] Lustre: Skipped 1 previous similar message [ 4077.073300] Lustre: lustre-MDT0001: new disk, initializing [ 4077.156978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4077.187573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4077.197366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4081.058176] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4088.706191] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4092.980185] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4098.350341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4098.570362] Lustre: lustre-OST0000: new disk, initializing [ 4098.574530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4098.579381] Lustre: 161047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4098.639260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.057529] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4108.821072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4108.834647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4108.942951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4110.284343] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4113.139944] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4114.926867] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:00:41 (1789185641) [ 4317.666378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4317.676885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4317.690241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.687809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.688220] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.695643] Lustre: Skipped 1 previous similar message [ 4318.708591] Lustre: Skipped 1 previous similar message [ 4322.153104] Lustre: server umount lustre-MDT0000 complete [ 4323.810536] LustreError: 159089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.834955] LustreError: 159089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4325.193733] LustreError: 159069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185853 with bad export cookie 17064239649674014954 [ 4325.194446] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.201619] LustreError: 159069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4328.927613] LustreError: 161708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.927680] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.960907] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4328.966440] Lustre: Skipped 1 previous similar message [ 4330.976057] LustreError: 160412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.976079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4331.472556] Lustre: server umount lustre-MDT0001 complete [ 4340.602707] Lustre: server umount lustre-OST0000 complete [ 4347.616429] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4354.734886] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4380.393823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4380.555892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4380.593845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4380.659360] Lustre: lustre-MDT0000: new disk, initializing [ 4380.710350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.725945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4383.696844] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4391.906667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.979966] Lustre: 165710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4392.020860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4392.026203] Lustre: Skipped 1 previous similar message [ 4392.108341] Lustre: lustre-MDT0001: new disk, initializing [ 4392.181764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4392.211540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4392.219340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4395.342925] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4402.638315] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4406.760820] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4411.625721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.821730] Lustre: lustre-OST0000: new disk, initializing [ 4411.825693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4411.831681] Lustre: 167676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4411.892670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.493152] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4418.591292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4418.600988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4418.663425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4423.963905] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4427.574441] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4427.725374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4430.988459] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4431.168465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4433.891161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4433.893000] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.893834] LustreError: Skipped 1 previous similar message [ 4433.900132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4433.901544] Lustre: Skipped 1 previous similar message [ 4433.921248] Lustre: Skipped 1 previous similar message [ 4438.658647] Lustre: server umount lustre-OST0000 complete [ 4443.617471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4443.623754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.635943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4447.526888] Lustre: server umount lustre-MDT0000 complete [ 4449.249099] LustreError: 165720:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4449.258459] LustreError: 165720:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4450.340838] LustreError: 165702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185978 with bad export cookie 17064239649674173784 [ 4450.351529] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.355898] LustreError: 165702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4450.667909] Lustre: server umount lustre-MDT0001 complete [ 4457.239076] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:06:23 (1789185983) [ 4469.774740] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4477.125342] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4503.557937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.744121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4503.765339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4503.837705] Lustre: lustre-MDT0000: new disk, initializing [ 4503.907127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.927735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4507.349419] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4514.505367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4514.601389] Lustre: 172428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4514.792096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4517.518771] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4524.605439] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4529.000368] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4533.816876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4534.096866] Lustre: 174406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4538.286462] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4541.499664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4541.514139] Lustre: Skipped 1 previous similar message [ 4541.517979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4541.646084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4545.825955] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4552.168613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.171480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.173713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4552.173720] Lustre: Skipped 2 previous similar messages [ 4552.176092] LustreError: Skipped 1 previous similar message [ 4552.227440] Lustre: Skipped 2 previous similar messages [ 4557.403681] Lustre: server umount lustre-OST0000 complete [ 4560.864869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4566.160048] Lustre: server umount lustre-MDT0000 complete [ 4567.524549] LustreError: 172441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.537422] LustreError: 172441:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4568.911523] LustreError: 172421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186097 with bad export cookie 17064239649674174666 [ 4568.914778] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4568.920826] LustreError: 172421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4569.165164] Lustre: server umount lustre-MDT0001 complete [ 4574.775100] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:08:21 (1789186101) [ 4584.897869] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4592.277921] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4617.031237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.208860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4617.233691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4617.237633] Lustre: Skipped 3 previous similar messages [ 4617.310723] Lustre: lustre-MDT0000: new disk, initializing [ 4617.313750] Lustre: Skipped 2 previous similar messages [ 4617.391948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.401496] Lustre: Skipped 2 previous similar messages [ 4617.412153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4620.677904] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4628.055556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.159899] Lustre: 178721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4628.365155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4631.403288] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4638.372779] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4642.275653] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4647.329729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4647.561153] Lustre: 180699:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4652.967911] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4655.807189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4660.947033] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4671.456969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4671.460248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4671.460837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4671.472830] Lustre: Skipped 3 previous similar messages [ 4671.481306] Lustre: Skipped 6 previous similar messages [ 4673.165584] Lustre: server umount lustre-OST0000 complete [ 4690.399131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4690.618614] Lustre: server umount lustre-MDT0000 complete [ 4691.940280] LustreError: 178728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.964853] LustreError: 178728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4693.435826] LustreError: 180697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186222 with bad export cookie 17064239649674175751 [ 4693.437317] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.454436] LustreError: 180697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4693.697714] Lustre: server umount lustre-MDT0001 complete [ 4701.289595] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4703.491967] Key type lgssc unregistered [ 4703.733318] LNet: 182557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4703.738830] LNetError: 182557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4703.753451] LNet: Removed LNI 192.168.206.115@tcp [ 4704.368616] Key type .llcrypt unregistered [ 4704.371538] Key type ._llcrypt unregistered [ 4721.395509] Key type ._llcrypt registered [ 4721.398383] Key type .llcrypt registered [ 4721.491898] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4722.312809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4722.344550] alg: No test for adler32 (adler32-zlib) [ 4723.372245] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 4723.582761] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4725.247195] Key type lgssc registered [ 4725.982827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4731.395767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4731.408927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.751852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4735.969898] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4741.580368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.821193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4744.685522] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4749.713769] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4753.243872] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4756.934308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4757.141929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4758.194615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4760.611753] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4765.056558] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4783.077451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.085974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4783.095161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4783.109317] Lustre: Skipped 1 previous similar message [ 4787.331801] Lustre: server umount lustre-OST0000 complete [ 4793.312726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.317875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4793.319656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4793.319952] Lustre: Skipped 2 previous similar messages [ 4796.363948] Lustre: server umount lustre-MDT0000 complete [ 4798.432621] LustreError: 184676:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4798.441476] LustreError: 184676:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4798.928735] LustreError: 184683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186327 with bad export cookie 15343625494189883153 [ 4798.933361] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4798.938524] LustreError: 184683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4799.150180] Lustre: server umount lustre-MDT0001 complete [ 4807.186866] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4809.655586] Key type lgssc unregistered [ 4809.893125] LNet: 187503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4809.903135] LNetError: 187503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4809.918016] LNet: Removed LNI 192.168.206.115@tcp [ 4810.568901] Key type .llcrypt unregistered [ 4810.570851] Key type ._llcrypt unregistered [ 4828.337241] Key type ._llcrypt registered [ 4828.338911] Key type .llcrypt registered [ 4828.421692] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4829.264917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4829.280158] alg: No test for adler32 (adler32-zlib) [ 4830.310834] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 4830.519909] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4832.167243] Key type lgssc registered [ 4832.985674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4838.130832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4838.141447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4839.392775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.456532] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4848.433664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.731445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4852.206588] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4857.365540] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4861.444949] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4866.184613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4866.480957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4871.327444] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4876.806678] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4884.916449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4894.013390] Lustre: lustre-OST0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 4894.176939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4894.182480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.202737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4895.204591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.206135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4895.206141] Lustre: Skipped 1 previous similar message [ 4898.436556] Lustre: server umount lustre-OST0000 complete [ 4903.800638] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a4b07713000.ost_server_uuid 50 [ 4904.897219] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4919.778950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4919.781398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4919.792190] Lustre: Skipped 1 previous similar message [ 4919.799408] Lustre: Skipped 2 previous similar messages [ 4923.335366] Lustre: server umount lustre-MDT0000 complete [ 4925.921271] LustreError: 188889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.935330] LustreError: 188889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4925.953024] LustreError: 190736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186454 with bad export cookie 15664359841976679409 [ 4925.953704] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4925.961576] LustreError: 190736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4926.186384] Lustre: server umount lustre-MDT0001 complete [ 4932.274936] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:14:19 (1789186459) [ 4937.560814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4937.970301] LustreError: 192824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.050057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4941.689640] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4943.335224] LustreError: 192825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4943.348116] LustreError: 192825:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4947.424655] LustreError: 192824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.213566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.503423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4951.897578] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4957.416600] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4961.683532] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4966.341379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4966.638328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.721905] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4977.572797] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4982.675149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4982.901247] Lustre: lustre-OST0001: new disk, initializing [ 4982.903815] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4982.909323] Lustre: 195665:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4983.005894] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4984.048227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4988.034884] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4990.519883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4990.530767] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4990.639256] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4994.845265] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4999.652095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4999.653123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.664084] LustreError: Skipped 1 previous similar message [ 4999.677089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4999.683172] Lustre: Skipped 3 previous similar messages [ 5002.576491] Lustre: server umount lustre-OST0000 complete [ 5005.618698] LustreError: 194644:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.746925] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5006.876675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5009.388382] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5009.533102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5015.851169] LustreError: 195742:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.864994] LustreError: 195742:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5021.668244] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5021.668708] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.676434] LustreError: Skipped 1 previous similar message [ 5021.690372] Lustre: Skipped 1 previous similar message [ 5021.694060] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5021.703292] Lustre: Skipped 4 previous similar messages [ 5027.367290] Lustre: server umount lustre-OST0001 complete [ 5030.369789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5030.377319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.390991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5030.398900] Lustre: Skipped 2 previous similar messages [ 5035.978590] Lustre: server umount lustre-MDT0000 complete [ 5037.025782] LustreError: 192824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5037.044420] LustreError: 192824:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 5038.863262] LustreError: 192804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186567 with bad export cookie 15664359841976680480 [ 5038.867026] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5038.869702] LustreError: 192804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5039.157923] Lustre: server umount lustre-MDT0001 complete [ 5056.236806] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 00:16:23 (1789186583) [ 5061.000724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.203866] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5061.230886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5061.390185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.774752] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5071.278108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.365673] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5071.400402] Lustre: 199103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5074.400038] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5080.171350] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5084.284555] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5089.050412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5089.182892] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5093.615090] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5096.543514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5099.212971] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5103.884070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5103.940164] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5108.262266] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5112.902532] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5116.390025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.392792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5116.398370] Lustre: Skipped 2 previous similar messages [ 5116.405894] Lustre: Skipped 2 previous similar messages [ 5120.656667] Lustre: server umount lustre-OST0000 complete [ 5121.510143] LustreError: 200954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.528965] LustreError: 200954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5136.870105] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5136.884720] LustreError: Skipped 1 previous similar message [ 5140.508900] Lustre: server umount lustre-OST0001 complete [ 5149.436430] Lustre: server umount lustre-MDT0000 complete [ 5151.767375] LustreError: 199094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186680 with bad export cookie 15664359841976681880 [ 5151.768607] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5151.775996] LustreError: 199094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5151.978156] Lustre: server umount lustre-MDT0001 complete [ 5166.425252] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:18:13 (1789186693) [ 5176.319891] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 5182.037700] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5203.980785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.158405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5204.177587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5204.227540] Lustre: lustre-MDT0000: new disk, initializing [ 5204.271582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5204.276874] Lustre: Skipped 3 previous similar messages [ 5204.287485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5207.466272] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5214.849793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.950699] Lustre: 207189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5215.003427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5215.010085] Lustre: Skipped 1 previous similar message [ 5215.082960] Lustre: lustre-MDT0001: new disk, initializing [ 5215.158556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5215.168052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5218.315987] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5224.986533] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5229.216956] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5233.578861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5233.746859] Lustre: lustre-OST0000: new disk, initializing [ 5233.749940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5233.755168] Lustre: 209157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5237.521554] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5242.901819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5242.909951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5242.994742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5243.257887] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5246.308160] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5246.447958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5249.183029] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5249.317490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5253.089797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5253.092281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5253.096015] LustreError: Skipped 1 previous similar message [ 5253.096962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5253.096966] Lustre: Skipped 6 previous similar messages [ 5253.111987] Lustre: Skipped 5 previous similar messages [ 5256.567669] Lustre: server umount lustre-OST0000 complete [ 5273.055765] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5273.158082] Lustre: server umount lustre-MDT0000 complete [ 5275.092115] LustreError: 207992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186803 with bad export cookie 15664359841976683441 [ 5275.092908] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.097609] LustreError: 207992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5275.254741] Lustre: server umount lustre-MDT0001 complete [ 5279.941965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.176486] LustreError: 211371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.187276] LustreError: 211371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5282.387682] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5287.351056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.279214] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5294.382182] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5297.347320] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5300.962638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5304.299289] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5307.910777] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5310.151371] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5310.274211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5312.390098] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5313.535081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 5315.559857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5315.568896] LustreError: Skipped 1 previous similar message [ 5317.602760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5317.604049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.606026] Lustre: Skipped 9 previous similar messages [ 5317.615124] Lustre: Skipped 3 previous similar messages [ 5320.566338] Lustre: server umount lustre-OST0000 complete [ 5323.664412] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5325.715508] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5351.046312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5351.259746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5351.265031] Lustre: Skipped 5 previous similar messages [ 5354.667179] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5357.414485] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.427508] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5358.615748] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5367.657595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.669205] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5373.506771] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 13 sec [ 5382.009706] Lustre: server umount lustre-OST0000 complete [ 5385.184193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5385.193468] LustreError: Skipped 3 previous similar messages [ 5390.689113] Lustre: server umount lustre-MDT0000 complete [ 5393.166873] LustreError: 211352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186921 with bad export cookie 15664359841976684323 [ 5393.170151] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5393.173236] LustreError: 211352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5393.379614] Lustre: server umount lustre-MDT0001 complete [ 5398.519500] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:22:05 (1789186925) [ 5402.589074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.613117] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5408.228464] LustreError: 217361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.236719] LustreError: 217361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5410.425488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.943727] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5417.074874] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5419.819543] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5423.189660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.647581] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5431.298072] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.046653] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5434.196596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5436.848925] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5436.989437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5440.502640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5440.611707] Lustre: lustre-OST0001: new disk, initializing [ 5440.616430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5440.620360] Lustre: 220727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5443.864258] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5449.054753] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5450.777294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5450.785579] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5450.843767] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5451.525680] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5451.647148] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5454.554946] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5454.719824] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5455.841714] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.845774] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5455.849736] Lustre: Skipped 5 previous similar messages [ 5455.852694] Lustre: Skipped 5 previous similar messages [ 5457.944365] Lustre: server umount lustre-OST0001 complete [ 5461.919728] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5462.049370] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5464.967727] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5465.148455] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5490.028909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5493.315492] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5495.273701] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5495.280095] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5495.280692] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5495.287808] Lustre: Skipped 1 previous similar message [ 5497.107407] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5497.912864] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5513.183113] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5513.228377] Lustre: server umount lustre-OST0001 complete [ 5528.033533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5528.040041] LustreError: Skipped 4 previous similar messages [ 5532.541526] Lustre: server umount lustre-OST0000 complete [ 5549.023104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5549.132171] Lustre: server umount lustre-MDT0000 complete [ 5551.189917] LustreError: 217343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187079 with bad export cookie 15664359841976685275 [ 5551.191286] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5551.198692] LustreError: 217343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5557.407070] Lustre: server umount lustre-MDT0001 complete [ 5570.320304] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 00:24:57 (1789187097) [ 5573.819162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.976795] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5574.003135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5576.289414] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5580.388369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.461382] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5580.480771] Lustre: 225960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5582.638406] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5586.361116] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5589.127600] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5592.498964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5592.603086] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5595.639943] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5599.069341] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5609.226061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5609.266890] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5609.372460] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5609.377415] Lustre: Skipped 8 previous similar messages [ 5612.159828] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5615.988333] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5617.945854] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5620.167925] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5620.300757] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5622.317817] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5622.421833] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5624.795378] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a4b034a3800.ost_server_uuid 50 [ 5625.411162] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a4b034a3800.ost_server_uuid in FULL state after 0 sec [ 5625.851065] Lustre: Permanently deactivating lustre-OST0001 [ 5625.858924] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5627.121632] Lustre: Permanently reactivating lustre-OST0001 [ 5634.575199] Lustre: server umount lustre-OST0001 complete [ 5642.413670] Lustre: server umount lustre-OST0000 complete [ 5650.412625] Lustre: server umount lustre-MDT0000 complete [ 5652.310328] LustreError: 225954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187180 with bad export cookie 15664359841976686549 [ 5652.311983] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5652.316979] LustreError: 225954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5652.486937] Lustre: server umount lustre-MDT0001 complete [ 5663.043885] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 00:26:30 (1789187190) [ 5666.652549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5666.798484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5666.817715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5666.823500] Lustre: Skipped 5 previous similar messages [ 5668.944769] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5673.138038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.204353] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5673.225960] Lustre: 232654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5675.225758] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5678.481572] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5681.168287] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5684.529841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5684.640263] Lustre: Permanently deactivating lustre-OST0000 [ 5684.648581] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5687.510207] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5690.784636] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5691.899155] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5693.811635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5696.504711] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5699.897415] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5701.685296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5706.074246] Lustre: Permanently reactivating lustre-OST0000 [ 5706.080869] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5706.084225] Lustre: Skipped 2 previous similar messages [ 5706.891537] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5706.897122] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.903221] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5706.922504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.934697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5711.845800] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.852522] Lustre: Skipped 14 previous similar messages [ 5711.856157] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5711.858800] Lustre: Skipped 30 previous similar messages [ 5716.880698] Lustre: server umount lustre-OST0001 complete [ 5716.962875] LustreError: 234497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5716.969448] LustreError: 234497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 28 previous similar messages [ 5724.581256] Lustre: server umount lustre-OST0000 complete [ 5732.530584] Lustre: server umount lustre-MDT0000 complete [ 5734.316577] LustreError: 232645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187262 with bad export cookie 15664359841976688208 [ 5734.319794] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.324917] LustreError: 232645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5734.506038] Lustre: server umount lustre-MDT0001 complete [ 5739.131305] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5740.551420] Key type lgssc unregistered [ 5740.748537] LNet: 237504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5740.752425] LNetError: 237504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5740.766430] LNet: Removed LNI 192.168.206.115@tcp [ 5741.179148] Key type .llcrypt unregistered [ 5741.180906] Key type ._llcrypt unregistered [ 5752.632984] Key type ._llcrypt registered [ 5752.634367] Key type .llcrypt registered [ 5752.692161] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5753.139755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5753.198870] alg: No test for adler32 (adler32-zlib) [ 5754.131935] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 5754.274191] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5755.871382] Key type lgssc registered [ 5756.381080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5760.716685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5760.724397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.926354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5763.452159] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5766.837274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.886182] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5766.902094] Lustre: 238932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5766.908186] Lustre: Permanently deactivating lustre-MDT0001 [ 5766.914925] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5767.047349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5768.622773] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5769.328712] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5769.333128] LustreError: 238973:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5769.338759] LustreError: 238973:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5769.343720] LustreError: 238973:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 5771.735355] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5774.202517] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5777.219688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5777.315547] LustreError: 240782:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.324815] LustreError: 240782:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5777.393513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.830990] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5782.501619] LustreError: 240783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5782.505500] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5782.511477] LustreError: 240783:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5782.909099] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5785.827422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5785.933034] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5786.985302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5786.986094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5786.989550] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5788.315960] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5791.287390] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5793.050506] Lustre: Permanently reactivating lustre-MDT0001 [ 5793.059212] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5793.061627] Lustre: Skipped 2 previous similar messages [ 5797.678975] Lustre: lustre-MDT0001: Client fb3be73c-8c92-4e9c-894a-93d7430281fd (at 192.168.206.15@tcp) reconnecting [ 5800.483166] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5801.199376] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5803.625157] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5803.720644] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5805.520394] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5805.628183] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5806.774718] Lustre: Permanently deactivating lustre-MDT0001 [ 5806.785075] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5806.789989] Lustre: Skipped 2 previous similar messages [ 5806.852375] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5816.801475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5816.803190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5816.806162] Lustre: Skipped 1 previous similar message [ 5818.150186] Lustre: server umount lustre-MDT0000 complete [ 5819.887658] LustreError: 242487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187348 with bad export cookie 16406886560969998519 [ 5819.890477] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.896711] LustreError: 242487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5820.037453] Lustre: server umount lustre-MDT0001 complete [ 5821.923743] Lustre: server umount lustre-OST0000 complete [ 5827.468095] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:29:14 (1789187354) [ 5832.411795] Lustre: server umount lustre-OST0001 complete [ 5835.419304] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 5839.433780] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5855.121651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.235819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5855.239293] Lustre: Skipped 2 previous similar messages [ 5855.251967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5855.297525] Lustre: lustre-MDT0000: new disk, initializing [ 5855.338285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.347575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5857.309258] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5862.551816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.598994] Lustre: 247189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5862.627273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5862.630070] Lustre: Skipped 1 previous similar message [ 5862.668733] Lustre: lustre-MDT0001: new disk, initializing [ 5862.717683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5862.722594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5864.477719] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5868.942299] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5871.215726] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5873.862201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5873.977168] Lustre: lustre-OST0000: new disk, initializing [ 5873.979968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5873.984076] Lustre: 249158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5874.025254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5874.028553] Lustre: Skipped 1 previous similar message [ 5876.293417] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5878.794132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5878.802873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5878.842971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5880.717493] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5881.584119] LustreError: 249165:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5886.392353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5886.466904] Lustre: lustre-OST0001: new disk, initializing [ 5886.469494] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5886.472905] Lustre: 250270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5888.860703] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5891.679284] LustreError: 249165:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5891.700930] LustreError: 247223:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5893.148390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5893.155599] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5893.194975] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5893.465433] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5894.134995] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5901.799177] LustreError: 247223:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5903.155637] Lustre: lustre-OST0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 5903.160244] Lustre: Skipped 3 previous similar messages [ 5903.333543] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.334216] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5903.344723] Lustre: Skipped 2 previous similar messages [ 5908.261859] Lustre: lustre-OST0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 5908.265359] Lustre: Skipped 2 previous similar messages [ 5908.747854] Lustre: server umount lustre-OST0001 complete [ 5913.571575] LustreError: 249143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5913.577933] LustreError: 249143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5918.690067] LustreError: 249142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.697805] LustreError: 249142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5923.809580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5923.812169] LustreError: 249142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5923.813660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5923.813665] Lustre: Skipped 2 previous similar messages [ 5923.817250] Lustre: Skipped 1 previous similar message [ 5923.834055] LustreError: 249142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5927.771307] Lustre: server umount lustre-OST0000 complete [ 5929.440797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5929.443583] LustreError: Skipped 1 previous similar message [ 5929.445935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.453966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5929.456707] Lustre: Skipped 1 previous similar message [ 5934.049337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5935.601905] Lustre: server umount lustre-MDT0000 complete [ 5937.083367] LustreError: 247180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187465 with bad export cookie 16406886560970001326 [ 5937.083823] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5937.089537] LustreError: 247180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5943.298789] Lustre: server umount lustre-MDT0001 complete [ 5947.277117] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5948.387201] Key type lgssc unregistered [ 5948.532482] LNet: 252304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5948.535932] LNetError: 252304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5948.544134] LNet: Removed LNI 192.168.206.115@tcp [ 5948.881139] Key type .llcrypt unregistered [ 5948.882380] Key type ._llcrypt unregistered [ 5965.229296] Key type ._llcrypt registered [ 5965.230937] Key type .llcrypt registered [ 5965.279547] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5965.659158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5965.697065] alg: No test for adler32 (adler32-zlib) [ 5966.568728] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 5966.665418] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5968.255204] Key type lgssc registered [ 5968.714611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5971.984078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5971.994742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.122320] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5973.133474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5973.207555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5975.012841] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5978.552794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5978.614949] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5978.628180] Lustre: 254498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5978.740485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5980.423104] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5983.556520] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5985.890979] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5988.846910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.931450] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5989.025151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.526294] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5994.785439] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5997.094372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6043.110695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6043.110837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6043.113245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6043.115658] LustreError: Skipped 1 previous similar message [ 6043.127429] Lustre: Skipped 1 previous similar message [ 6047.389234] Lustre: server umount lustre-OST0000 complete [ 6049.105028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6053.895932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6054.036977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.205103] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6059.277813] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6063.269939] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.276382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.276693] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6063.285610] Lustre: Skipped 1 previous similar message [ 6094.304580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6094.306424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6094.308313] LustreError: Skipped 1 previous similar message [ 6094.317126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6094.321180] Lustre: Skipped 2 previous similar messages [ 6099.192838] Lustre: server umount lustre-OST0000 complete [ 6101.984141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6101.988837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.994395] Lustre: Skipped 1 previous similar message [ 6101.999030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6104.547259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6104.547439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6109.665281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6109.668697] Lustre: Skipped 2 previous similar messages [ 6115.295108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6115.375740] Lustre: server umount lustre-MDT0000 complete [ 6117.347536] LustreError: 254490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187645 with bad export cookie 18135998437103927859 [ 6117.350091] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.352422] LustreError: 254490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6117.535595] Lustre: server umount lustre-MDT0001 complete [ 6122.687749] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6124.039971] Key type lgssc unregistered [ 6124.192622] LNet: 259911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6124.197980] LNetError: 259911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6124.205579] LNet: Removed LNI 192.168.206.115@tcp [ 6124.611438] Key type .llcrypt unregistered [ 6124.613391] Key type ._llcrypt unregistered [ 6136.323649] Key type ._llcrypt registered [ 6136.325264] Key type .llcrypt registered [ 6136.379160] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6136.814200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6136.858100] alg: No test for adler32 (adler32-zlib) [ 6137.748695] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 6137.869917] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6139.471205] Key type lgssc registered [ 6139.877224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6143.073915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6143.082881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.264544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6145.866321] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6149.262167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.470291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6151.208601] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6154.203911] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6156.369372] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6159.054225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6159.206436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6161.502205] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6164.597085] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6168.423987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6180.833722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6180.839193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.849235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6183.905965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.907363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6183.907429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6186.218389] Lustre: server umount lustre-OST0000 complete [ 6189.026455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6189.036490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6189.041405] Lustre: Skipped 2 previous similar messages [ 6194.144094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6194.147940] Lustre: Skipped 1 previous similar message [ 6199.264681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.269555] Lustre: Skipped 1 previous similar message [ 6202.335355] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6202.394740] Lustre: server umount lustre-MDT0000 complete [ 6203.820335] LustreError: 261281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187732 with bad export cookie 156684245254121483 [ 6203.823588] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.825351] LustreError: 261281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6203.952302] Lustre: server umount lustre-MDT0001 complete [ 6208.188533] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6209.412213] Key type lgssc unregistered [ 6209.548407] LNet: 265601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6209.551512] LNetError: 265601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6209.558343] LNet: Removed LNI 192.168.206.115@tcp [ 6209.896646] Key type .llcrypt unregistered [ 6209.898293] Key type ._llcrypt unregistered [ 6217.447954] Key type ._llcrypt registered [ 6217.449672] Key type .llcrypt registered [ 6217.498323] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6217.910885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6217.944087] alg: No test for adler32 (adler32-zlib) [ 6218.824097] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 6218.922514] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6220.511310] Key type lgssc registered [ 6220.943408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6223.861692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6223.871015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.052110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6226.575444] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6229.803401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6229.965293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6231.608270] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6234.365638] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6236.583373] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6239.207937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6239.345054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6241.498981] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6244.570328] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6249.578892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6255.073489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6255.074822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.078269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.081178] Lustre: Skipped 1 previous similar message [ 6256.218220] Lustre: server umount lustre-OST0000 complete [ 6259.867013] Lustre: server umount lustre-MDT0000 complete [ 6260.196328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.198535] LustreError: 266792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6260.201886] Lustre: Skipped 1 previous similar message [ 6260.210890] LustreError: 266792:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6261.339421] LustreError: 266777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187789 with bad export cookie 17880506148950916804 [ 6261.340913] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6261.344454] LustreError: 266777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6261.479536] Lustre: server umount lustre-MDT0001 complete [ 6265.652473] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6266.836211] Key type lgssc unregistered [ 6267.001473] LNet: 270473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6267.005324] LNetError: 270473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6267.015309] LNet: Removed LNI 192.168.206.115@tcp [ 6267.387606] Key type .llcrypt unregistered [ 6267.390496] Key type ._llcrypt unregistered [ 6277.780580] Key type ._llcrypt registered [ 6277.782822] Key type .llcrypt registered [ 6277.836388] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6278.194859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6278.202305] alg: No test for adler32 (adler32-zlib) [ 6279.095902] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 6279.208745] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6280.799355] Key type lgssc registered [ 6281.261482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6284.425781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6284.436371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6285.645948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6287.365015] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6290.863343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.026615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6292.668253] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6295.827783] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6298.079246] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6300.945201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6301.116481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6303.520040] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6306.701550] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6312.359099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6327.776395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6327.778859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.781142] LustreError: Skipped 1 previous similar message [ 6327.782987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.791890] Lustre: Skipped 1 previous similar message [ 6332.898933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6332.902130] Lustre: Skipped 2 previous similar messages [ 6333.033677] Lustre: server umount lustre-OST0000 complete [ 6336.992293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6336.995313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.000411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6338.017390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.136177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6343.138772] Lustre: Skipped 3 previous similar messages [ 6348.256480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6348.260935] Lustre: Skipped 1 previous similar message [ 6349.279241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6349.354165] Lustre: server umount lustre-MDT0000 complete [ 6350.767406] LustreError: 271843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187879 with bad export cookie 17101024869291286155 [ 6350.772140] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6350.772501] LustreError: 271843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6350.890759] Lustre: server umount lustre-MDT0001 complete [ 6355.379780] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6356.884579] Key type lgssc unregistered [ 6357.054923] LNet: 276184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6357.058373] LNetError: 276184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6357.074801] LNet: Removed LNI 192.168.206.115@tcp [ 6357.475817] Key type .llcrypt unregistered [ 6357.478130] Key type ._llcrypt unregistered [ 6365.699951] Key type ._llcrypt registered [ 6365.701572] Key type .llcrypt registered [ 6365.746893] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6366.152122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6366.198138] alg: No test for adler32 (adler32-zlib) [ 6367.088787] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 6367.211452] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6368.807938] Key type lgssc registered [ 6369.305235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6372.848936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6372.865155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.113641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6375.984442] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6380.038158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6380.258559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6382.257450] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6385.729863] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6388.146675] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6391.155851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.307319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6393.721191] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6396.778967] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6398.503926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6413.792671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6413.796730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6413.802871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.820746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.821667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.822385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.824290] Lustre: Skipped 1 previous similar message [ 6417.888466] Lustre: server umount lustre-OST0000 complete [ 6419.936646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.939422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6419.950159] Lustre: Skipped 1 previous similar message [ 6419.957323] Lustre: Skipped 1 previous similar message [ 6425.055979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6425.059962] Lustre: Skipped 1 previous similar message [ 6425.776329] Lustre: server umount lustre-MDT0000 complete [ 6427.470764] LustreError: 280360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187956 with bad export cookie 2483121362708454872 [ 6427.474303] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6427.477423] LustreError: 280360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6427.653088] Lustre: server umount lustre-MDT0001 complete [ 6432.377644] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6433.725842] Key type lgssc unregistered [ 6433.887807] LNet: 281109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6433.892291] LNetError: 281109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6433.909635] LNet: Removed LNI 192.168.206.115@tcp [ 6434.296593] Key type .llcrypt unregistered [ 6434.298315] Key type ._llcrypt unregistered [ 6439.118398] Key type ._llcrypt registered [ 6439.119580] Key type .llcrypt registered [ 6439.174922] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6446.941691] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 6454.104577] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6454.570490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6454.592723] alg: No test for adler32 (adler32-zlib) [ 6455.453393] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 6455.559263] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6457.151237] Key type lgssc registered [ 6457.643042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6470.991832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6471.002507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.098610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6472.112170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6472.155261] Lustre: lustre-MDT0000: new disk, initializing [ 6472.184890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6472.192853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6473.844221] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6478.868246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.926577] Lustre: 284631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6478.968490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6478.972125] Lustre: Skipped 1 previous similar message [ 6479.009510] Lustre: lustre-MDT0001: new disk, initializing [ 6479.061906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6479.081782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6479.090158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6480.802133] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6485.243901] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6487.388165] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6490.118202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6490.241062] Lustre: lustre-OST0000: new disk, initializing [ 6490.243854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6490.248025] Lustre: 286597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6490.285289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6492.641042] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6496.791924] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6497.796825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6497.801209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6497.855931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6498.550939] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6498.648788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6500.379333] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6500.476495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6502.883506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.883958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6502.884950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6502.888578] Lustre: Skipped 1 previous similar message [ 6502.892108] LustreError: Skipped 1 previous similar message [ 6507.425671] Lustre: server umount lustre-OST0000 complete [ 6510.049094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6510.052893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.060539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6510.063205] Lustre: Skipped 1 previous similar message [ 6513.120365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6513.120794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.240060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6518.242250] Lustre: Skipped 2 previous similar messages [ 6523.359278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6523.360861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6523.366764] Lustre: Skipped 1 previous similar message [ 6523.428436] Lustre: server umount lustre-MDT0000 complete [ 6524.822277] LustreError: 285424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188053 with bad export cookie 7414568449519146423 [ 6524.825249] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6524.827588] LustreError: 285424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6524.943659] Lustre: server umount lustre-MDT0001 complete [ 6528.121709] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:40:55 (1789188055) [ 6530.694034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6530.888660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6532.423680] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6535.653231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.800351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6537.386458] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6540.054225] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6542.194129] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6544.908028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6545.073425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.516737] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6550.424256] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6566.882065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6566.882254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6566.883614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6566.892793] Lustre: Skipped 1 previous similar message [ 6570.262101] Lustre: server umount lustre-OST0000 complete [ 6572.000486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.048954] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6574.051655] LustreError: Skipped 1 previous similar message [ 6577.879674] Lustre: server umount lustre-MDT0000 complete [ 6579.231121] LustreError: 291609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188107 with bad export cookie 7414568449519147305 [ 6579.233336] LustreError: 289744:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6579.234073] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6579.234724] LustreError: 291609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6579.350806] Lustre: server umount lustre-MDT0001 complete [ 6583.264302] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6584.469783] Key type lgssc unregistered [ 6584.607551] LNet: 292561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6584.611277] LNetError: 292561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6584.622542] LNet: Removed LNI 192.168.206.115@tcp [ 6584.973263] Key type .llcrypt unregistered [ 6584.974714] Key type ._llcrypt unregistered [ 6594.172847] Key type ._llcrypt registered [ 6594.174384] Key type .llcrypt registered [ 6594.220211] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 6600.893811] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6601.332409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6601.358617] alg: No test for adler32 (adler32-zlib) [ 6602.227932] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 6602.334552] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6603.919218] Key type lgssc registered [ 6604.336746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6620.773986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6620.783741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.869463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6621.879016] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6621.915302] Lustre: lustre-MDT0000: new disk, initializing [ 6621.939753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.946458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6623.461938] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6627.930372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.984061] Lustre: 296354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6628.014853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6628.017475] Lustre: Skipped 1 previous similar message [ 6628.057025] Lustre: lustre-MDT0001: new disk, initializing [ 6628.094795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6628.108273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6628.111696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6629.562320] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6633.488585] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6635.475634] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6637.964097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6638.072421] Lustre: lustre-OST2710: new disk, initializing [ 6638.074698] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6638.078149] Lustre: 298322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6638.113821] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6640.119662] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6644.029958] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6646.688871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.755178] Lustre: lustre-OST03e8: new disk, initializing [ 6646.757622] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6646.761322] Lustre: 299374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6646.801675] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6647.304983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6647.310772] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6647.316025] Lustre: Skipped 1 previous similar message [ 6647.362708] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6647.366090] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6648.928091] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6651.633816] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6654.436252] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6654.516188] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6656.015374] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6656.095301] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6657.578926] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6657.654260] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6659.197848] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6659.277716] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6662.629539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.631096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6662.634019] Lustre: Skipped 2 previous similar messages [ 6662.635933] Lustre: Skipped 1 previous similar message [ 6666.465470] Lustre: server umount lustre-MDT0000 complete [ 6667.948789] LustreError: 296347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188196 with bad export cookie 16142265425894705148 [ 6667.950896] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6667.955977] LustreError: 296347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6668.081523] Lustre: server umount lustre-MDT0001 complete [ 6680.076709] Lustre: server umount lustre-OST2710 complete [ 6691.825562] Lustre: server umount lustre-OST03e8 complete [ 6698.895375] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 6702.455363] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6715.967316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.059803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6716.072620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6716.110763] Lustre: lustre-MDT0000: new disk, initializing [ 6716.138210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6716.149684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6716.154182] Lustre: Skipped 1 previous similar message [ 6717.672498] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6722.244438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.296055] Lustre: 304622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6722.408471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6722.413888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6723.973149] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6727.852439] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6729.710555] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6732.118492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6732.221663] Lustre: lustre-OST0000: new disk, initializing [ 6732.223291] Lustre: Skipped 1 previous similar message [ 6732.225284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6732.227952] Lustre: Skipped 2 previous similar messages [ 6732.230437] Lustre: 306591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6732.261030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.263719] Lustre: Skipped 1 previous similar message [ 6734.070456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6734.074280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6734.110620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6734.330662] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6738.016079] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6739.394932] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6739.469608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6740.864114] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6740.944478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6744.545623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6744.546530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6744.547828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6744.547834] Lustre: Skipped 2 previous similar messages [ 6744.548927] LustreError: Skipped 1 previous similar message [ 6744.559976] Lustre: Skipped 2 previous similar messages [ 6747.676709] Lustre: server umount lustre-OST0000 complete [ 6749.664940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6749.666837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6749.670288] Lustre: Skipped 1 previous similar message [ 6749.672633] Lustre: Skipped 1 previous similar message [ 6754.784545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6754.787242] Lustre: Skipped 2 previous similar messages [ 6755.159538] Lustre: server umount lustre-MDT0000 complete [ 6756.479124] LustreError: 304616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188285 with bad export cookie 16142265425894706730 [ 6756.482071] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6756.483592] LustreError: 304616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6756.598454] Lustre: server umount lustre-MDT0001 complete [ 6759.608077] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:44:46 (1789188286) [ 6760.122540] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6760.750837] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:44:48 (1789188288) [ 6761.858794] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6771.228443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.317957] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6771.324841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6771.380273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.814844] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6775.589135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6775.626738] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6775.637317] Lustre: 310441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6777.182926] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6793.059536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6793.134265] Lustre: Denying initial registration attempt from nid 192.168.206.115@tcp, specified as failover [ 6793.137818] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6793.142544] LustreError: 311889:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 6793.145751] LustreError: 311889:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6793.148564] LustreError: 311889:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6793.177222] Lustre: server umount lustre-OST0000 complete [ 6793.179279] LustreError: 311889:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6796.256589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6796.256965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.264052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6796.266301] Lustre: Skipped 1 previous similar message [ 6801.495349] Lustre: server umount lustre-MDT0000 complete [ 6802.753076] LustreError: 310432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188331 with bad export cookie 16142265425894707598 [ 6802.756677] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6802.760474] LustreError: 310432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6802.862285] Lustre: server umount lustre-MDT0001 complete [ 6806.465533] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6807.504748] Key type lgssc unregistered [ 6807.633351] LNet: 313013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6807.635514] LNetError: 313013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6807.646248] LNet: Removed LNI 192.168.206.115@tcp [ 6807.939563] Key type .llcrypt unregistered [ 6807.940534] Key type ._llcrypt unregistered [ 6811.524049] Key type ._llcrypt registered [ 6811.524907] Key type .llcrypt registered [ 6811.569439] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6812.036817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6812.065147] alg: No test for adler32 (adler32-zlib) [ 6812.938420] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 6813.037137] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6814.623245] Key type lgssc registered [ 6814.996760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6839.006994] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6842.000933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6842.009785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.094412] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6843.104780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6843.164703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6844.592101] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6847.565372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6847.607526] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6847.620527] Lustre: 315748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6847.722466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6849.187247] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6851.741450] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6853.588804] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6869.090084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6869.152080] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6869.210619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6871.140284] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6873.634855] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6874.417724] Lustre: server umount lustre-OST0000 complete [ 6876.656832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.658046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6876.661263] Lustre: Skipped 1 previous similar message [ 6881.760262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6881.763217] Lustre: Skipped 2 previous similar messages [ 6881.807184] Lustre: server umount lustre-MDT0000 complete [ 6882.913146] LustreError: 315737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188411 with bad export cookie 5061262781179898043 [ 6882.915913] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6882.917413] LustreError: 315737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6883.023317] Lustre: server umount lustre-MDT0001 complete [ 6886.366256] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6887.365916] Key type lgssc unregistered [ 6887.509378] LNet: 319308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6887.512583] LNetError: 319308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6887.519330] LNet: Removed LNI 192.168.206.115@tcp [ 6887.817940] Key type .llcrypt unregistered [ 6887.819170] Key type ._llcrypt unregistered [ 6896.482391] Key type ._llcrypt registered [ 6896.483363] Key type .llcrypt registered [ 6896.519889] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6896.828136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6896.881779] alg: No test for adler32 (adler32-zlib) [ 6897.732432] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 6897.820030] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6899.391121] Key type lgssc registered [ 6899.717590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6902.162587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6902.168749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6903.305987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6904.560449] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6907.141150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.257021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6908.569422] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6910.890537] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6912.646304] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6916.897441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6917.003208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.766263] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6921.032507] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6937.571037] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6937.571787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6937.574600] Lustre: Skipped 2 previous similar messages [ 6937.577844] Lustre: Skipped 2 previous similar messages [ 6940.751175] Lustre: server umount lustre-MDT0000 complete [ 6941.879855] LustreError: 320680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188470 with bad export cookie 7108298746127576874 [ 6941.882949] LustreError: 320680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6941.883261] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6942.687884] LustreError: 322279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.688068] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6942.691734] LustreError: 322279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6942.695337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6942.698816] Lustre: Skipped 1 previous similar message [ 6946.719747] LustreError: 323186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.719979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6946.724751] LustreError: 323186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6946.729366] Lustre: Skipped 1 previous similar message [ 6948.065432] Lustre: server umount lustre-MDT0001 complete [ 6950.132789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6952.935086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.263809] LustreError: 320007:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001cc2b348 x1876100043006208/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6967.370304] LustreError: 322527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.392945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6967.416039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6968.777450] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6971.489037] LustreError: 324307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.745304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.870765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6973.295257] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6975.770500] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6976.996508] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6978.310897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6979.968129] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6983.524087] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6987.235127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.235230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.239165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6987.242532] Lustre: Skipped 1 previous similar message [ 6992.353262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6992.355069] Lustre: Skipped 2 previous similar messages [ 6999.007092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6999.064221] Lustre: server umount lustre-OST0000 complete [ 7002.592592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7002.592890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.593543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7002.593546] Lustre: Skipped 2 previous similar messages [ 7002.600486] Lustre: Skipped 1 previous similar message [ 7006.350452] Lustre: server umount lustre-MDT0000 complete [ 7007.460814] LustreError: 322539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188536 with bad export cookie 7108298746127584028 [ 7007.463820] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7007.463896] LustreError: 322539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7007.564839] Lustre: server umount lustre-MDT0001 complete [ 7010.639420] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7011.631877] Key type lgssc unregistered [ 7011.746296] LNet: 326977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7011.748237] LNetError: 326977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7011.755245] LNet: Removed LNI 192.168.206.115@tcp [ 7012.030171] Key type .llcrypt unregistered [ 7012.031170] Key type ._llcrypt unregistered [ 7020.425720] Key type ._llcrypt registered [ 7020.426823] Key type .llcrypt registered [ 7020.464163] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7020.804227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7020.814827] alg: No test for adler32 (adler32-zlib) [ 7021.670598] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 7021.764038] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7023.351169] Key type lgssc registered [ 7023.747853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7026.308029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7026.313932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.453561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7028.842114] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7031.745411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.878594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7033.312444] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7035.625425] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7037.257738] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7042.015821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7042.018070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.022398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7043.040225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7043.041027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.307995] Lustre: server umount lustre-MDT0000 complete [ 7045.563129] LustreError: 329100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188574 with bad export cookie 13488522300137566608 [ 7045.566291] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7045.570959] LustreError: 329100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7045.688224] Lustre: server umount lustre-MDT0001 complete [ 7050.538213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7050.638650] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7050.649915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7050.724335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7052.089171] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7055.051035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7055.086804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7055.097026] Lustre: 330951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7055.195778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7056.641594] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7059.150628] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7060.908022] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7065.251962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7065.323275] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7065.326836] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7065.331785] LustreError: 332832:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 7065.334323] LustreError: 332832:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7065.337309] LustreError: 332832:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7065.362270] Lustre: server umount lustre-OST0000 complete [ 7065.363600] LustreError: 332832:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7070.306378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7070.377778] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7070.450664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.387546] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7074.860824] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7077.483948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7077.517104] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7077.542779] Lustre: lustre-OST0001: new disk, initializing [ 7077.544731] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7077.547932] Lustre: 334361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7077.696243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7077.700250] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7077.724545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7077.749670] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7079.368419] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7081.547864] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7082.979405] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7082.979650] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.980841] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7082.980846] Lustre: Skipped 1 previous similar message [ 7082.982956] LustreError: Skipped 1 previous similar message [ 7082.986727] Lustre: Skipped 1 previous similar message [ 7084.540856] Lustre: server umount lustre-OST0001 complete [ 7088.098276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7088.098832] LustreError: 333298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.099126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7088.099906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7088.099910] Lustre: Skipped 1 previous similar message [ 7088.101657] LustreError: Skipped 1 previous similar message [ 7088.106550] LustreError: 333298:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7093.218052] LustreError: 333299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7093.218067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7093.222721] LustreError: 333299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7093.224257] Lustre: Skipped 2 previous similar messages [ 7098.339091] LustreError: 333299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.343709] LustreError: 333299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7099.871121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7099.926647] Lustre: server umount lustre-OST0000 complete [ 7101.407795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7101.410103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.413955] Lustre: Skipped 1 previous similar message [ 7101.415950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.417958] Lustre: Skipped 2 previous similar messages [ 7107.354518] Lustre: server umount lustre-MDT0000 complete [ 7108.576949] LustreError: 330961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7108.581854] LustreError: 330961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7108.591111] LustreError: 330942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188637 with bad export cookie 13488522300137567147 [ 7108.594611] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7108.595329] LustreError: 330942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7108.710818] Lustre: server umount lustre-MDT0001 complete [ 7112.503837] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7113.599041] Key type lgssc unregistered [ 7113.735416] LNet: 336322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7113.737872] LNetError: 336322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7113.749339] LNet: Removed LNI 192.168.206.115@tcp [ 7114.037114] Key type .llcrypt unregistered [ 7114.038051] Key type ._llcrypt unregistered [ 7143.893242] Key type ._llcrypt registered [ 7143.894157] Key type .llcrypt registered [ 7143.928881] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7149.818308] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7150.179699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7150.198348] alg: No test for adler32 (adler32-zlib) [ 7151.046031] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 7151.127654] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7152.711222] Key type lgssc registered [ 7153.029769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7163.233930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7163.238836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7164.310108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7164.319307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7164.351880] Lustre: lustre-MDT0000: new disk, initializing [ 7164.369119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7164.375253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7165.643405] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7169.582874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7169.623511] Lustre: 341218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7169.644723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7169.647576] Lustre: Skipped 1 previous similar message [ 7169.681920] Lustre: lustre-MDT0001: new disk, initializing [ 7169.710455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7169.721739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7169.725169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7170.978208] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7174.513076] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7176.305571] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7178.526722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7178.633263] Lustre: lustre-OST0000: new disk, initializing [ 7178.637459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7178.640271] Lustre: 343183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7178.669957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7180.526404] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7184.159085] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7185.548990] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7187.443997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7187.447639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7187.478399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7187.658772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7188.974641] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7189.045144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7192.546178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7192.546267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7192.550395] LustreError: Skipped 1 previous similar message [ 7192.555726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7192.559346] Lustre: Skipped 1 previous similar message [ 7195.796420] Lustre: server umount lustre-OST0000 complete [ 7197.664306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7197.665090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7197.668175] Lustre: Skipped 2 previous similar messages [ 7202.784306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7202.786289] Lustre: Skipped 2 previous similar messages [ 7203.096133] Lustre: server umount lustre-MDT0000 complete [ 7204.332121] LustreError: 342014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188732 with bad export cookie 16554198691956147446 [ 7204.335914] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7204.336831] LustreError: 342014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7204.456020] Lustre: server umount lustre-MDT0001 complete [ 7207.312258] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:52:14 (1789188734) [ 7209.526970] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:52:16 (1789188736) [ 7212.028541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7212.181888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7213.364989] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7215.931159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.318545] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7219.462767] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7221.100773] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7223.098731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7223.195848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7223.197761] Lustre: Skipped 1 previous similar message [ 7224.901138] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7227.074209] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7240.814759] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7241.369981] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7241.916639] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7242.488185] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7243.176063] Lustre: Failing over lustre-MDT0000 [ 7243.335903] Lustre: server umount lustre-MDT0000 complete [ 7246.819217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.820132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7246.821600] LustreError: 348568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.821624] LustreError: 348568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7246.824159] Lustre: Skipped 2 previous similar messages [ 7247.655670] LustreError: 346688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.661947] LustreError: 346688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7251.936338] LustreError: 347435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.941877] LustreError: 347435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7256.014135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.049656] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7256.129543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.160531] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7257.410469] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7257.888825] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7261.156445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7261.165693] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7261.181847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7262.858091] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7263.393982] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7265.181142] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7265.835534] Lustre: Failing over lustre-MDT0000 [ 7265.982697] Lustre: server umount lustre-MDT0000 complete [ 7266.274827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7266.276237] LustreError: 345953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7266.280436] Lustre: Skipped 2 previous similar messages [ 7266.288074] LustreError: 345953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7266.290544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7271.391977] LustreError: 347435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.396613] LustreError: 347435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7278.785423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7278.827291] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7278.909992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7278.931941] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7280.199313] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7283.488982] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7284.196210] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7284.197789] Lustre: Skipped 2 previous similar messages [ 7284.205260] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7284.220641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7285.817588] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7286.295808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7289.313217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.314508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7289.316165] Lustre: Skipped 2 previous similar messages [ 7289.316367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7289.317546] Lustre: Skipped 2 previous similar messages [ 7294.433445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7294.435240] Lustre: Skipped 2 previous similar messages [ 7294.492254] Lustre: server umount lustre-MDT0000 complete [ 7295.600726] LustreError: 345978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188824 with bad export cookie 16554198691956150127 [ 7295.601931] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7295.605923] LustreError: 345978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7295.707375] Lustre: server umount lustre-MDT0001 complete [ 7307.093310] Lustre: server umount lustre-OST0000 complete [ 7312.126608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.270374] LustreError: 352303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.277396] LustreError: 352303:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7312.292384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.605853] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7316.474844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.034120] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7320.452559] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7322.265166] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7324.630273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7326.675281] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7329.218833] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7337.063042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7342.562073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7342.562366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7342.564443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7342.566245] LustreError: Skipped 1 previous similar message [ 7342.572636] Lustre: Skipped 1 previous similar message [ 7347.150536] Lustre: server umount lustre-OST0000 complete [ 7352.800422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7354.522698] Lustre: server umount lustre-MDT0000 complete [ 7355.738420] LustreError: 354803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188884 with bad export cookie 16554198691956150848 [ 7355.741673] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7355.741963] LustreError: 354803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7355.872220] Lustre: server umount lustre-MDT0001 complete [ 7359.654595] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7360.732357] Key type lgssc unregistered [ 7360.865365] LNet: 355855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7360.867735] LNetError: 355855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7360.874323] LNet: Removed LNI 192.168.206.115@tcp [ 7361.201191] Key type .llcrypt unregistered [ 7361.202316] Key type ._llcrypt unregistered [ 7371.001967] Key type ._llcrypt registered [ 7371.003470] Key type .llcrypt registered [ 7371.043396] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7371.501975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7371.510412] alg: No test for adler32 (adler32-zlib) [ 7372.377306] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 7372.477127] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7374.071255] Key type lgssc registered [ 7374.464812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7377.023406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7377.030859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.182637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7379.561232] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7382.412486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.543794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7383.944359] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7386.241915] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7388.035980] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7390.318299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7390.445164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7392.366685] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7394.788249] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7403.750942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7413.216284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7413.218555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.223488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7414.241453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7414.241860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7415.441895] Lustre: server umount lustre-MDT0000 complete [ 7416.584460] LustreError: 357334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188945 with bad export cookie 10654383903015517189 [ 7416.589388] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7416.702770] Lustre: server umount lustre-MDT0001 complete [ 7428.119051] Lustre: server umount lustre-OST0000 complete [ 7432.585140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.762352] LustreError: 361081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7432.768881] LustreError: 361081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7432.790954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7434.248252] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7437.148155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.269835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7438.703913] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7441.127542] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7443.053618] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7445.430057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.564507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7447.660748] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7450.064684] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7457.891258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7467.999737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7468.002131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7468.006058] Lustre: Skipped 1 previous similar message [ 7468.007973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7468.009719] Lustre: Skipped 2 previous similar messages [ 7473.487106] Lustre: server umount lustre-MDT0000 complete [ 7473.632820] LustreError: 361081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7473.637606] LustreError: 361081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7474.624129] LustreError: 361062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189003 with bad export cookie 10654383903015518288 [ 7474.627323] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7474.629923] LustreError: 361062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7474.727810] Lustre: server umount lustre-MDT0001 complete [ 7486.102751] Lustre: server umount lustre-OST0000 complete [ 7490.876153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7491.021063] LustreError: 365010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.043882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.292942] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7494.903054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7496.171353] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7498.411031] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7500.079681] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7502.109091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7503.964884] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7506.226590] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7515.559053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7520.739137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7520.742859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.747442] Lustre: Skipped 2 previous similar messages [ 7520.749855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.752773] Lustre: Skipped 4 previous similar messages [ 7524.308632] Lustre: server umount lustre-OST0000 complete [ 7525.857192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.857431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7525.858173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7525.862268] Lustre: Skipped 2 previous similar messages [ 7525.867556] LustreError: Skipped 1 previous similar message [ 7531.680901] Lustre: server umount lustre-MDT0000 complete [ 7532.962242] LustreError: 364990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189061 with bad export cookie 10654383903015519296 [ 7532.965455] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7532.966316] LustreError: 364990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7533.082865] Lustre: server umount lustre-MDT0001 complete [ 7537.116585] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7538.144655] Key type lgssc unregistered [ 7538.272487] LNet: 368561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7538.276380] LNetError: 368561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7538.285453] LNet: Removed LNI 192.168.206.115@tcp [ 7538.625487] Key type .llcrypt unregistered [ 7538.627277] Key type ._llcrypt unregistered [ 7548.344912] Key type ._llcrypt registered [ 7548.345859] Key type .llcrypt registered [ 7548.385265] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7548.749083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7548.765788] alg: No test for adler32 (adler32-zlib) [ 7549.611403] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 7549.695621] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7551.271123] Key type lgssc registered [ 7551.592359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7553.965978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7553.972098] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.975319] LustreError: 369990:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7553.983105] LustreError: 369990:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7553.985059] LustreError: 369990:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7553.987086] LustreError: 369990:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7553.990061] LustreError: 369990:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7556.310037] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7556.313735] LustreError: 370386:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7556.322197] LustreError: 370386:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7556.324439] LustreError: 370386:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7556.327111] LustreError: 370386:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7556.330147] LustreError: 370386:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7562.351098] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7563.313814] Key type lgssc unregistered [ 7563.443301] LNet: 371386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7563.446276] LNetError: 371386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7563.457249] LNet: Removed LNI 192.168.206.115@tcp [ 7563.727552] Key type .llcrypt unregistered [ 7563.729178] Key type ._llcrypt unregistered [ 7569.061491] Key type ._llcrypt registered [ 7569.062373] Key type .llcrypt registered [ 7569.094731] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7574.854720] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7575.153105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7575.204086] alg: No test for adler32 (adler32-zlib) [ 7576.055988] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 7576.144840] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7577.735187] Key type lgssc registered [ 7578.094358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7589.697083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7589.703135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7590.788133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7590.797890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7590.836157] Lustre: lustre-MDT0000: new disk, initializing [ 7590.859529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.865888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7592.285707] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7596.242898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.283691] Lustre: 374510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7596.304365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7596.306263] Lustre: Skipped 1 previous similar message [ 7596.339264] Lustre: lustre-MDT0001: new disk, initializing [ 7596.367270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7596.377855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7596.381103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7597.620667] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7601.080105] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7602.719524] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7604.740144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7604.832978] Lustre: lustre-OST0000: new disk, initializing [ 7604.834430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7604.836633] Lustre: 376478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7604.862902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7606.639300] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7610.310911] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7611.651731] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7613.942617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7613.945661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7613.974607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7614.770827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7615.942507] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7616.003149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7619.042169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.042355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7619.046197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.047371] LustreError: Skipped 1 previous similar message [ 7619.051779] Lustre: Skipped 1 previous similar message [ 7622.675604] Lustre: server umount lustre-OST0000 complete [ 7624.160877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.161432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7624.165195] Lustre: Skipped 2 previous similar messages [ 7629.280078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7629.283079] Lustre: Skipped 2 previous similar messages [ 7629.978250] Lustre: server umount lustre-MDT0000 complete [ 7631.264154] LustreError: 375307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189159 with bad export cookie 2561426767900777159 [ 7631.267497] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7631.268336] LustreError: 375307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7631.378848] Lustre: server umount lustre-MDT0001 complete [ 7634.109568] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7634.647426] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:59:21 (1789189161) [ 7636.768444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.943055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7638.249450] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7641.061741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7641.187354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7642.583583] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7644.906436] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7646.683360] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7648.895180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7650.864846] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7653.319539] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7655.608618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7655.675505] Lustre: lustre-OST0001: new disk, initializing [ 7655.677282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7655.679681] Lustre: 381788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7655.710768] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7655.712668] Lustre: Skipped 1 previous similar message [ 7657.575150] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7660.040413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7660.043611] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7660.070979] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7661.313906] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7665.120101] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7665.122163] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.126485] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7665.128423] Lustre: Skipped 1 previous similar message [ 7669.243876] Lustre: server umount lustre-OST0001 complete [ 7673.315236] LustreError: 381162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.321293] LustreError: 381162:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7678.436403] LustreError: 380777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7678.440438] LustreError: 380777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7682.016058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7682.018202] LustreError: Skipped 1 previous similar message [ 7682.019264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.022103] Lustre: Skipped 1 previous similar message [ 7682.023511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7682.025379] Lustre: Skipped 3 previous similar messages [ 7683.554809] LustreError: 381162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.558658] LustreError: 381162:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7688.673797] LustreError: 381162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.678134] LustreError: 381162:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7693.794618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7693.795917] LustreError: 381162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.796091] Lustre: Skipped 5 previous similar messages [ 7693.800062] LustreError: 381162:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7695.327106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7695.372071] Lustre: server umount lustre-OST0000 complete [ 7697.375688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7697.377905] LustreError: Skipped 1 previous similar message [ 7697.379265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.382743] Lustre: Skipped 1 previous similar message [ 7702.675956] Lustre: server umount lustre-MDT0000 complete [ 7703.916488] LustreError: 378936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189232 with bad export cookie 2561426767900778027 [ 7703.919568] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7703.920680] LustreError: 378936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7718.367095] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7718.449862] Lustre: server umount lustre-MDT0001 complete [ 7722.131603] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7723.208239] Key type lgssc unregistered [ 7723.337315] LNet: 383773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7723.339563] LNetError: 383773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7723.348252] LNet: Removed LNI 192.168.206.115@tcp [ 7723.639540] Key type .llcrypt unregistered [ 7723.640801] Key type ._llcrypt unregistered [ 7740.476695] Key type ._llcrypt registered [ 7740.477784] Key type .llcrypt registered [ 7740.513959] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7740.818702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7740.857058] alg: No test for adler32 (adler32-zlib) [ 7741.714148] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 7741.804293] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7743.383172] Key type lgssc registered [ 7743.731113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7746.217634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7746.223505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.304477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7747.312306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7747.367397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7748.681868] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7751.378766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7751.410141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7751.420408] Lustre: 386556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7751.514816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7752.824473] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7755.089603] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7756.816460] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7761.376487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.377294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.383271] Lustre: Skipped 1 previous similar message [ 7763.604986] Lustre: server umount lustre-MDT0000 complete [ 7764.906618] LustreError: 386547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189293 with bad export cookie 6489474434723177164 [ 7764.909897] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7764.913632] LustreError: 386547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7765.044343] Lustre: server umount lustre-MDT0001 complete [ 7767.972049] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:01:35 (1789189295) [ 7769.956268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7770.099309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7771.263327] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7773.898255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7774.019978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7775.419635] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7777.789264] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7779.537225] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7781.722941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7781.790166] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7781.853261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7783.931024] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7786.154770] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7789.693040] Lustre: Permanently deactivating lustre-OST0000 [ 7789.697134] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7792.112944] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7792.766618] LustreError: 391951:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7792.771522] LustreError: 391951:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7792.774803] LustreError: 391951:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7793.306447] LustreError: 392000:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7793.308125] LustreError: 392000:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7800.012829] Lustre: server umount lustre-OST0000 complete [ 7800.938345] LustreError: 392249:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7800.941242] LustreError: 392249:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7803.360783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7803.361377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7803.364928] Lustre: Skipped 1 previous similar message [ 7803.367171] Lustre: Skipped 1 previous similar message [ 7808.480200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7808.482518] Lustre: Skipped 1 previous similar message [ 7811.577103] Lustre: server umount lustre-MDT0000 complete [ 7812.832309] LustreError: 389045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189341 with bad export cookie 6489474434723177731 [ 7812.835203] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.836176] LustreError: 389045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7823.379379] Lustre: server umount lustre-MDT0001 complete [ 7826.235834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7827.571605] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7828.450922] LustreError: 393408:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7830.500280] Lustre: 393746:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.115@tcp [ 7831.180906] Lustre: server umount MGS complete [ 7834.940745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.093122] LustreError: 394470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7835.098039] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7835.099560] LustreError: 394470:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7835.101191] Lustre: Skipped 1 previous similar message [ 7835.119442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.458952] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7839.302795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7839.406385] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7840.871786] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7843.296550] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7845.139543] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7845.784896] Lustre: Permanently reactivating lustre-OST0000 [ 7845.789416] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7845.791587] Lustre: Skipped 2 previous similar messages [ 7868.895160] LustreError: 396154:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7871.327723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7871.445703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7871.447729] Lustre: Skipped 1 previous similar message [ 7873.398257] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7875.736466] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7888.865263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7888.867163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.868651] LustreError: Skipped 1 previous similar message [ 7888.869330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7888.874895] Lustre: Skipped 1 previous similar message [ 7893.457459] Lustre: server umount lustre-OST0000 complete [ 7896.031953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7896.034611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7896.038867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7896.042109] Lustre: Skipped 1 previous similar message [ 7904.224438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.227392] Lustre: Skipped 3 previous similar messages [ 7909.343161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7909.407459] Lustre: server umount lustre-MDT0000 complete [ 7910.638226] LustreError: 395207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189439 with bad export cookie 6489474434723178760 [ 7910.641548] LustreError: 395207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7910.641811] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7910.747297] Lustre: server umount lustre-MDT0001 complete [ 7914.437748] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7915.523285] Key type lgssc unregistered [ 7915.652361] LNet: 399043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7915.654857] LNetError: 399043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7915.664376] LNet: Removed LNI 192.168.206.115@tcp [ 7915.995443] Key type .llcrypt unregistered [ 7915.997370] Key type ._llcrypt unregistered [ 7922.047481] Key type ._llcrypt registered [ 7922.048509] Key type .llcrypt registered [ 7922.086814] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7928.326660] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7928.675728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7928.760795] alg: No test for adler32 (adler32-zlib) [ 7929.623674] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 7929.705893] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7931.287169] Key type lgssc registered [ 7931.665161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7943.073582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7943.079860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7944.160040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7944.170169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7944.207351] Lustre: lustre-MDT0000: new disk, initializing [ 7944.229713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7944.237661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7945.662677] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7949.700738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.746294] Lustre: 402168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7949.768985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7949.770724] Lustre: Skipped 1 previous similar message [ 7949.808475] Lustre: lustre-MDT0001: new disk, initializing [ 7949.843686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7949.854956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7949.859304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7951.204797] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7954.857154] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7956.654799] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7959.035637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7959.138252] Lustre: lustre-OST0000: new disk, initializing [ 7959.139882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7959.144273] Lustre: 404136:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7959.174810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7961.028911] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7961.783222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7961.787050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7961.818133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7964.649986] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7966.040796] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7966.122272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7967.522987] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7967.598782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7972.322080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.322630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7972.328160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7972.329994] LustreError: Skipped 1 previous similar message [ 7972.331938] Lustre: Skipped 1 previous similar message [ 7974.352529] Lustre: server umount lustre-OST0000 complete [ 7977.440779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7977.441785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7977.446481] Lustre: Skipped 2 previous similar messages [ 7981.790020] Lustre: server umount lustre-MDT0000 complete [ 7983.023319] LustreError: 402964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189511 with bad export cookie 16047145441603425270 [ 7983.026844] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7983.026854] LustreError: 402964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7983.149291] Lustre: server umount lustre-MDT0001 complete [ 7985.778795] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7986.312783] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:05:13 (1789189513) [ 7988.554793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.694031] LustreError: 406591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7988.700953] LustreError: 406591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7988.715837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7990.083925] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7992.810471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.921779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7994.251490] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7996.667196] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7998.449531] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8000.701030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.746865] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8005.155248] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8005.828030] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8016.755938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8018.400440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8018.402764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.406739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8018.409995] Lustre: Skipped 1 previous similar message [ 8019.427451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.276391] Lustre: server umount lustre-OST0000 complete [ 8028.639753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8028.642063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.645295] Lustre: Skipped 1 previous similar message [ 8028.647116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8028.648885] Lustre: Skipped 2 previous similar messages [ 8031.699833] Lustre: server umount lustre-MDT0000 complete [ 8032.924221] LustreError: 406574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189561 with bad export cookie 16047145441603426138 [ 8032.927225] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8032.929325] LustreError: 406574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8033.050861] Lustre: server umount lustre-MDT0001 complete [ 8036.655406] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8037.754165] Key type lgssc unregistered [ 8037.894361] LNet: 410330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8037.896576] LNetError: 410330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8037.908424] LNet: Removed LNI 192.168.206.115@tcp [ 8038.256344] Key type .llcrypt unregistered [ 8038.257645] Key type ._llcrypt unregistered [ 8045.693467] Key type ._llcrypt registered [ 8045.694491] Key type .llcrypt registered [ 8045.729673] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8045.831322] Key type .llcrypt unregistered [ 8045.832368] Key type ._llcrypt unregistered [ 8052.755979] Key type ._llcrypt registered [ 8052.757176] Key type .llcrypt registered [ 8052.802456] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8053.324475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8053.343529] alg: No test for adler32 (adler32-zlib) [ 8054.201384] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 8054.292219] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8055.871227] Key type lgssc registered [ 8056.225549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8058.677922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8058.683535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8059.820133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8061.115029] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8063.838527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8063.958506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8065.782707] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8068.147937] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8069.088501] LustreError: 412508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.093385] LustreError: 412508:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8070.113574] LustreError: 412504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.117149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8070.118651] LustreError: 412504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8070.441025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.555633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8071.866235] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8080.351896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8080.354100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.361093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8080.865833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8080.868033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.871898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8084.814127] Lustre: server umount lustre-OST0000 complete [ 8090.081322] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8090.083749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8090.089584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8092.178732] Lustre: server umount lustre-MDT0000 complete [ 8093.421468] LustreError: 412488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189622 with bad export cookie 7547175019695071916 [ 8093.426233] LustreError: 412488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8093.427631] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8093.556811] Lustre: server umount lustre-MDT0001 complete [ 8097.237411] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8098.265739] Key type lgssc unregistered [ 8098.395373] LNet: 415654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8098.397812] LNetError: 415654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8098.404303] LNet: Removed LNI 192.168.206.115@tcp [ 8098.714243] Key type .llcrypt unregistered [ 8098.715168] Key type ._llcrypt unregistered [ 8107.756056] Key type ._llcrypt registered [ 8107.757732] Key type .llcrypt registered [ 8107.801694] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8108.275047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8108.333676] alg: No test for adler32 (adler32-zlib) [ 8109.206224] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 8109.301865] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8110.887213] Key type lgssc registered [ 8111.289833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8113.886337] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8113.893554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8129.311312] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8134.431269] LustreError: 417016:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.115@tcp: failed processing log, type 4: rc = -110 [ 8161.055160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8163.024993] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8165.318577] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8167.525062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8167.693219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8167.719506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8168.961162] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8171.849838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8171.982954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8173.323718] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8175.743553] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8177.569870] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8181.220585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.222807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8181.224235] Lustre: Skipped 1 previous similar message [ 8185.930640] Lustre: server umount lustre-OST0000 complete [ 8187.359776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8187.362031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.365801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8187.368349] Lustre: Skipped 1 previous similar message [ 8191.456298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8191.456956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8193.232840] Lustre: server umount lustre-MDT0000 complete [ 8194.451095] LustreError: 419549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189723 with bad export cookie 7136504648251946863 [ 8194.455298] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8194.455833] LustreError: 419549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8194.564114] Lustre: server umount lustre-MDT0001 complete [ 8198.282228] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8199.271065] Key type lgssc unregistered [ 8199.391491] LNet: 420586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8199.394252] LNetError: 420586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8199.403523] LNet: Removed LNI 192.168.206.115@tcp [ 8199.730538] Key type .llcrypt unregistered [ 8199.731722] Key type ._llcrypt unregistered [ 8208.660973] Key type ._llcrypt registered [ 8208.663098] Key type .llcrypt registered [ 8208.707804] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8209.026535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8209.064061] alg: No test for adler32 (adler32-zlib) [ 8209.918067] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 8210.004754] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8211.591218] Key type lgssc registered [ 8211.958650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8214.516692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8214.522770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.669812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8216.915568] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8219.705807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.835841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8221.122264] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8223.504236] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8225.280126] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8227.510367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8227.619566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.449777] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8231.827392] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8240.930612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8245.727864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8245.731179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8245.735124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.240487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.241886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.593380] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 8247.595638] Lustre: Skipped 2 previous similar messages [ 8249.496265] Lustre: server umount lustre-MDT0000 complete [ 8250.244856] LustreError: 421955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.15@tcp arrived at 1789189778 with bad export cookie 10617702210830479496 [ 8250.249303] LustreError: 421955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8250.886649] LustreError: 422708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189779 with bad export cookie 10617702210830478873 [ 8250.891129] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8250.891803] LustreError: 422708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8251.359931] LustreError: 424466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.360412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.364374] LustreError: 424466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8251.367476] Lustre: Skipped 1 previous similar message [ 8251.368013] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8255.392203] LustreError: 422714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8255.392568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8255.398783] LustreError: 422714:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8256.970987] Lustre: server umount lustre-OST0000 complete [ 8259.222724] Lustre: server umount lustre-MDT0001 complete [ 8262.909690] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8263.990619] Key type lgssc unregistered [ 8264.126514] LNet: 425676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8264.129143] LNetError: 425676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8264.138428] LNet: Removed LNI 192.168.206.115@tcp [ 8264.478053] Key type .llcrypt unregistered [ 8264.479716] Key type ._llcrypt unregistered [ 8273.443210] Key type ._llcrypt registered [ 8273.444268] Key type .llcrypt registered [ 8273.480264] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8273.768292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8273.816787] alg: No test for adler32 (adler32-zlib) [ 8274.663657] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 8274.754891] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8276.335138] Key type lgssc registered [ 8276.696604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8279.454343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8279.461383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8280.620169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8281.908976] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8284.474149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.588775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8285.788748] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8287.989756] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8289.719580] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.964129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8292.083339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.023463] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8296.578365] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8305.381328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8310.239793] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8310.242073] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.245781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8310.754767] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.758612] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8312.103367] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 8314.070418] Lustre: server umount lustre-MDT0001 complete [ 8315.872054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8315.872374] LustreError: 427065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8315.874991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.879715] LustreError: 427065:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8315.883996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.993217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.993230] LustreError: 427065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8321.743400] Lustre: server umount lustre-OST0000 complete [ 8323.047389] Lustre: server umount lustre-MDT0000 complete [ 8327.396772] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8328.364746] Key type lgssc unregistered [ 8328.486384] LNet: 430769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8328.489392] LNetError: 430769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8328.498404] LNet: Removed LNI 192.168.206.115@tcp [ 8328.817206] Key type .llcrypt unregistered [ 8328.818364] Key type ._llcrypt unregistered [ 8336.919798] Key type ._llcrypt registered [ 8336.920823] Key type .llcrypt registered [ 8336.957506] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 8343.409289] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8343.718157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8343.723921] alg: No test for adler32 (adler32-zlib) [ 8344.586300] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 8344.681167] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8346.255214] Key type lgssc registered [ 8346.684476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8359.675082] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8365.094117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8365.098828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.184575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8366.194073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8366.229639] Lustre: lustre-MDT0000: new disk, initializing [ 8366.248397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8366.253424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8367.635977] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8373.751343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8373.895938] Lustre: lustre-OST0000: new disk, initializing [ 8373.898526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8373.901156] Lustre: Skipped 1 previous similar message [ 8373.904159] Lustre: 435771:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8373.938568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8376.007974] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8379.568499] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8379.901618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8379.904278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8379.912076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8390.021178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.067560] Lustre: 434750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8390.091535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8390.127161] Lustre: lustre-MDT0001: new disk, initializing [ 8390.165996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8390.179342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8390.182956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8391.500301] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8394.959419] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8395.055323] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8396.547672] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8397.671205] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 8400.170609] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 8402.402082] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8402.404565] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.406149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8402.409340] Lustre: Skipped 1 previous similar message [ 8405.283476] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 8405.288930] Lustre: Skipped 1 previous similar message [ 8405.687578] Lustre: server umount lustre-MDT0001 complete [ 8412.641321] LustreError: 434758:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8412.647332] LustreError: 434758:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8417.763177] LustreError: 436482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8417.768702] LustreError: 436482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8420.832644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8420.836416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.843975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8422.881057] LustreError: 434763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.201403] Lustre: server umount lustre-OST0000 complete [ 8431.734406] Lustre: server umount lustre-MDT0000 complete [ 8436.309436] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8437.286579] Key type lgssc unregistered [ 8437.405346] LNet: 439520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8437.407586] LNetError: 439520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8437.417266] LNet: Removed LNI 192.168.206.115@tcp [ 8437.726271] Key type .llcrypt unregistered [ 8437.727952] Key type ._llcrypt unregistered [ 8450.472452] Key type ._llcrypt registered [ 8450.473387] Key type .llcrypt registered [ 8450.507162] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8450.601613] Key type .llcrypt unregistered [ 8450.603072] Key type ._llcrypt unregistered [ 8457.168996] Key type ._llcrypt registered [ 8457.170639] Key type .llcrypt registered [ 8457.208326] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8457.547636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8457.621802] alg: No test for adler32 (adler32-zlib) [ 8458.475054] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 8458.558251] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8460.143170] Key type lgssc registered [ 8460.478044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8470.403073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8470.415913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8471.519870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8471.532735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8471.581902] Lustre: lustre-MDT0000: new disk, initializing [ 8471.604201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8471.611700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8473.014699] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8477.092673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.134059] Lustre: 442871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8477.155884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8477.157581] Lustre: Skipped 1 previous similar message [ 8477.192848] Lustre: lustre-MDT0001: new disk, initializing [ 8477.223904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8477.236362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8477.239417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8478.674046] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8482.287613] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8484.250229] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8486.622642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8486.727662] Lustre: lustre-OST0000: new disk, initializing [ 8486.729770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8486.732951] Lustre: 444841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8486.762615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8489.213486] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8490.038089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8490.043363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8490.071061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8492.887918] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8497.631920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8497.634477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8497.638546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.192749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.193799] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.198468] Lustre: Skipped 1 previous similar message [ 8501.353144] Lustre: server umount lustre-MDT0000 complete [ 8502.836786] LustreError: 444839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190031 with bad export cookie 2750509584551659090 [ 8502.843813] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8502.845195] LustreError: 444839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8502.981915] Lustre: server umount lustre-MDT0001 complete [ 8513.476391] Lustre: server umount lustre-OST0000 complete [ 8517.112067] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:14:04 (1789190044) [ 8520.062324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8520.231612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8521.569580] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8524.382035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8524.495289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8525.810769] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8528.141519] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8529.906797] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8545.122082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8545.193458] Lustre: Found index 0 for lustre-OST0000, updating log [ 8545.271739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8547.260745] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8549.808577] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8565.736965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8575.970376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8575.972641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8575.975592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8575.975600] Lustre: Skipped 2 previous similar messages [ 8575.975732] LustreError: Skipped 1 previous similar message [ 8575.986038] Lustre: Skipped 1 previous similar message [ 8577.995468] Lustre: server umount lustre-OST0000 complete [ 8590.485536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.486390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8590.489035] Lustre: Skipped 1 previous similar message [ 8590.495039] Lustre: Skipped 4 previous similar messages [ 8595.936257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8595.938618] Lustre: Skipped 1 previous similar message [ 8596.651731] Lustre: server umount lustre-MDT0000 complete [ 8598.007601] LustreError: 446846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190126 with bad export cookie 2750509584551660469 [ 8598.014633] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8598.015236] LustreError: 446846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8598.147409] Lustre: server umount lustre-MDT0001 complete [ 8601.134261] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:15:28 (1789190128) [ 8607.071385] LNet: 1 local NIs in recovery (showing 1): 192.168.206.115@tcp [ 9181.917162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9182.101901] LustreError: 453285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9182.109942] LustreError: 453285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9182.134199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9183.770505] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9186.789401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.942840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9188.605102] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9191.301733] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9193.208205] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9208.548452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9208.682925] Lustre: Found index 0 for lustre-OST0000, updating log [ 9208.764730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9210.659471] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9213.114136] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9213.861527] Lustre: server umount lustre-OST0000 complete [ 9215.157135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.159259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9215.163091] Lustre: Skipped 1 previous similar message [ 9220.576214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.578868] Lustre: Skipped 2 previous similar messages [ 9221.349785] Lustre: server umount lustre-MDT0000 complete [ 9222.784089] LustreError: 453311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190751 with bad export cookie 2750509584551661540 [ 9222.787206] LustreError: 453311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9222.787591] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9222.894264] Lustre: server umount lustre-MDT0001 complete [ 9234.783317] LNet: 1 local NIs in recovery (showing 1): 192.168.206.115@tcp [ 9240.873280] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 9243.973898] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9257.169100] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:26:24 (1789190784) [ 9263.540931] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9264.754571] Key type lgssc unregistered [ 9271.071588] LNet: 459934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9271.075910] LNetError: 459934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9340.709466] LNet: Removed LNI 192.168.206.115@tcp [ 9341.087811] Key type .llcrypt unregistered [ 9341.088990] Key type ._llcrypt unregistered [ 9348.752168] Key type ._llcrypt registered [ 9348.753134] Key type .llcrypt registered [ 9348.793370] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9349.178883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9349.249526] alg: No test for adler32 (adler32-zlib) [ 9350.118210] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 9350.224407] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9351.815258] Key type lgssc registered [ 9352.179038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9353.617381] LNet: Added LNI 192.168.206.115@tcp100 [8/256/0/180] [ 9353.897259] LNet: Added LNI 192.168.206.115@tcp101 [8/256/0/180] [ 9354.174416] LNet: Added LNI 192.168.206.115@tcp102 [8/256/0/180] [ 9354.479275] LNet: Added LNI 192.168.206.115@tcp103 [8/256/0/180] [ 9354.765906] LNet: Added LNI 192.168.206.115@tcp104 [8/256/0/180] [ 9355.073307] LNet: Added LNI 192.168.206.115@tcp105 [8/256/0/180] [ 9355.343100] LNet: Added LNI 192.168.206.115@tcp106 [8/256/0/180] [ 9355.633132] LNet: Added LNI 192.168.206.115@tcp107 [8/256/0/180] [ 9355.963026] LNet: Added LNI 192.168.206.115@tcp108 [8/256/0/180] [ 9356.366583] LNet: Added LNI 192.168.206.115@tcp109 [8/256/0/180] [ 9356.776851] LNet: Added LNI 192.168.206.115@tcp110 [8/256/0/180] [ 9357.198111] LNet: Added LNI 192.168.206.115@tcp111 [8/256/0/180] [ 9357.589860] LNet: Added LNI 192.168.206.115@tcp112 [8/256/0/180] [ 9357.972465] LNet: Added LNI 192.168.206.115@tcp113 [8/256/0/180] [ 9358.300081] LNet: Added LNI 192.168.206.115@tcp114 [8/256/0/180] [ 9358.702931] LNet: Added LNI 192.168.206.115@tcp115 [8/256/0/180] [ 9359.030530] LNet: Added LNI 192.168.206.115@tcp116 [8/256/0/180] [ 9359.318512] LNet: Added LNI 192.168.206.115@tcp117 [8/256/0/180] [ 9359.670315] LNet: Added LNI 192.168.206.115@tcp118 [8/256/0/180] [ 9360.038952] LNet: Added LNI 192.168.206.115@tcp119 [8/256/0/180] [ 9360.454516] LNet: Added LNI 192.168.206.115@tcp120 [8/256/0/180] [ 9360.878496] LNet: Added LNI 192.168.206.115@tcp121 [8/256/0/180] [ 9361.211273] LNet: Added LNI 192.168.206.115@tcp122 [8/256/0/180] [ 9361.554596] LNet: Added LNI 192.168.206.115@tcp123 [8/256/0/180] [ 9361.869122] LNet: Added LNI 192.168.206.115@tcp124 [8/256/0/180] [ 9362.244984] LNet: Added LNI 192.168.206.115@tcp125 [8/256/0/180] [ 9362.593991] LNet: Added LNI 192.168.206.115@tcp126 [8/256/0/180] [ 9362.960803] LNet: Added LNI 192.168.206.115@tcp127 [8/256/0/180] [ 9363.365287] LNet: Added LNI 192.168.206.115@tcp128 [8/256/0/180] [ 9363.807913] LNet: Added LNI 192.168.206.115@tcp129 [8/256/0/180] [ 9364.262893] LNet: Added LNI 192.168.206.115@tcp130 [8/256/0/180] [ 9364.665670] LNet: Added LNI 192.168.206.115@tcp131 [8/256/0/180] [ 9365.037309] LNet: Added LNI 192.168.206.115@tcp132 [8/256/0/180] [ 9365.427670] LNet: Added LNI 192.168.206.115@tcp133 [8/256/0/180] [ 9365.807803] LNet: Added LNI 192.168.206.115@tcp134 [8/256/0/180] [ 9366.195291] LNet: Added LNI 192.168.206.115@tcp135 [8/256/0/180] [ 9368.835947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9368.847043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9369.903828] Lustre: 462928:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [ 9369.936971] LNetError: 462950:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 9369.950777] LNetError: Protocol error connecting to 192.168.206.115@tcp113 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 9369.954062] LNetError: Skipped 2 previous similar messages [ 9369.997281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9370.006743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9370.041728] Lustre: lustre-MDT0000: new disk, initializing [ 9370.062810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9370.067957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9370.719472] LNet: 6 local NIs in recovery (showing 6): 192.168.206.115@tcp127, 192.168.206.115@tcp124, 192.168.206.115@tcp125, 192.168.206.115@tcp128, 192.168.206.115@tcp129, 192.168.206.115@tcp126 [ 9370.723980] LNetError: 462958:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 9370.733885] LNetError: 462958:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 9370.738787] LNetError: Protocol error connecting to 192.168.206.115@tcp108 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 9370.743704] LNetError: Skipped 6 previous similar messages [ 9371.579980] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9372.770709] LNetError: 460393:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 9372.773566] LNetError: 460393:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 9372.777729] LNetError: Protocol error connecting to 192.168.206.115@tcp125 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 9372.780814] LNetError: Skipped 1 previous similar message [ 9376.334748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9376.387087] Lustre: 462946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9376.413018] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9376.414697] Lustre: Skipped 1 previous similar message [ 9376.451273] Lustre: lustre-MDT0001: new disk, initializing [ 9376.461963] Lustre: 463748:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [ 9376.464340] Lustre: 463748:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9376.500588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9376.511945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9376.516363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9376.864608] LNetError: 460390:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 9376.868213] LNetError: Protocol error connecting to 192.168.206.115@tcp125 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 9378.196519] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9382.397519] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9383.929312] Lustre: 464512:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [ 9383.933915] Lustre: 464512:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9384.032953] LNetError: 462948:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 9384.038844] LNetError: Protocol error connecting to 192.168.206.115@tcp125 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 9384.803837] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9387.623326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9387.794028] Lustre: lustre-OST0000: new disk, initializing [ 9387.796989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9387.801451] Lustre: 464921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9387.856785] Lustre: 464926:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [ 9387.860371] Lustre: 464926:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 9387.889537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9390.377352] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9394.170399] Lustre: 465582:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [ 9394.175894] Lustre: 465582:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9394.204264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9394.212445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9394.269602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9394.678850] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9399.266210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9399.266901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9399.270894] LustreError: Skipped 1 previous similar message [ 9399.271790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9399.278756] Lustre: Skipped 1 previous similar message [ 9399.283863] Lustre: Skipped 1 previous similar message [ 9399.391900] LNetError: 460391:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 9399.395236] LNetError: Protocol error connecting to 192.168.206.115@tcp113 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 9403.500791] Lustre: server umount lustre-OST0000 complete [ 9407.456093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9407.459091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9407.465772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9409.506127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9409.506504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9414.624250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9414.627039] Lustre: Skipped 2 previous similar messages [ 9416.799986] LNetError: 460393:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 9416.802739] LNetError: 460393:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 9416.804903] LNetError: Protocol error connecting to 192.168.206.115@tcp127 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 9416.807672] LNetError: Skipped 6 previous similar messages [ 9419.231087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9419.298560] Lustre: server umount lustre-MDT0000 complete [ 9420.704422] LustreError: 462937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190949 with bad export cookie 15599682204750909572 [ 9420.708299] LustreError: 462937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9420.708633] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9420.847419] Lustre: server umount lustre-MDT0001 complete [ 9424.571576] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9425.649135] Key type lgssc unregistered [ 9425.829160] LNet: 466834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9425.833559] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9425.836756] LNet: Removed LNI 192.168.206.115@tcp [ 9426.847174] LNet: Removed LNI 192.168.206.115@tcp100 [ 9426.850555] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9426.855151] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9426.860129] LNet: Removed LNI 192.168.206.115@tcp101 [ 9427.871157] LNet: Removed LNI 192.168.206.115@tcp102 [ 9427.872447] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9427.874364] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9427.876301] LNet: Removed LNI 192.168.206.115@tcp103 [ 9428.895198] LNet: Removed LNI 192.168.206.115@tcp104 [ 9429.919259] LNet: Removed LNI 192.168.206.115@tcp105 [ 9429.922208] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9429.927462] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9429.933092] LNet: Removed LNI 192.168.206.115@tcp106 [ 9430.943186] LNet: Removed LNI 192.168.206.115@tcp107 [ 9431.967149] LNet: Removed LNI 192.168.206.115@tcp108 [ 9432.991173] LNet: Removed LNI 192.168.206.115@tcp109 [ 9434.015108] LNet: Removed LNI 192.168.206.115@tcp110 [ 9434.017606] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9434.021197] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9434.029505] LNet: Removed LNI 192.168.206.115@tcp111 [ 9435.039121] LNet: Removed LNI 192.168.206.115@tcp112 [ 9436.063101] LNet: Removed LNI 192.168.206.115@tcp113 [ 9437.087100] LNet: Removed LNI 192.168.206.115@tcp114 [ 9438.111242] LNet: Removed LNI 192.168.206.115@tcp115 [ 9439.135120] LNet: Removed LNI 192.168.206.115@tcp116 [ 9440.159226] LNet: Removed LNI 192.168.206.115@tcp117 [ 9441.183143] LNet: Removed LNI 192.168.206.115@tcp118 [ 9442.207298] LNet: Removed LNI 192.168.206.115@tcp119 [ 9442.210091] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9442.214668] LNetError: 466834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 9442.219792] LNet: Removed LNI 192.168.206.115@tcp120 [ 9443.231198] LNet: Removed LNI 192.168.206.115@tcp121 [ 9444.255342] LNet: Removed LNI 192.168.206.115@tcp122 [ 9445.279115] LNet: Removed LNI 192.168.206.115@tcp123 [ 9446.303121] LNet: Removed LNI 192.168.206.115@tcp124 [ 9446.304381] LNet: Removed LNI 192.168.206.115@tcp125 [ 9447.327167] LNet: Removed LNI 192.168.206.115@tcp126 [ 9448.351130] LNet: Removed LNI 192.168.206.115@tcp127 [ 9449.375154] LNet: Removed LNI 192.168.206.115@tcp128 [ 9450.399139] LNet: Removed LNI 192.168.206.115@tcp129 [ 9450.400884] LNet: Removed LNI 192.168.206.115@tcp130 [ 9450.402080] LNet: Removed LNI 192.168.206.115@tcp131 [ 9450.403476] LNet: Removed LNI 192.168.206.115@tcp132 [ 9450.404863] LNet: Removed LNI 192.168.206.115@tcp133 [ 9450.406251] LNet: Removed LNI 192.168.206.115@tcp134 [ 9450.413632] LNet: Removed LNI 192.168.206.115@tcp135 [ 9450.744855] Key type .llcrypt unregistered [ 9450.747869] Key type ._llcrypt unregistered [ 9459.718555] Key type ._llcrypt registered [ 9459.720242] Key type .llcrypt registered [ 9459.784823] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 9466.639509] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9467.052022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9467.058151] alg: No test for adler32 (adler32-zlib) [ 9467.908990] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 9467.996071] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9469.583145] Key type lgssc registered [ 9469.927767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9484.336554] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:30:11 (1789191011) [ 9491.417387] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9492.397858] Key type lgssc unregistered [ 9492.525307] LNet: 470887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9492.528652] LNetError: 470887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9492.539316] LNet: Removed LNI 192.168.206.115@tcp [ 9492.861412] Key type .llcrypt unregistered [ 9492.862577] Key type ._llcrypt unregistered [ 9499.388967] Key type ._llcrypt registered [ 9499.390857] Key type .llcrypt registered [ 9499.455341] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9499.871690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9499.887740] alg: No test for adler32 (adler32-zlib) [ 9500.744704] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 9500.835453] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9502.415191] Key type lgssc registered [ 9502.798838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9506.694633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9506.699612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9507.779837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9507.789127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9507.828217] Lustre: lustre-MDT0000: new disk, initializing [ 9507.851169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.857622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9509.446309] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9514.946312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9515.016252] Lustre: 472072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9515.047693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9515.050373] Lustre: Skipped 1 previous similar message [ 9515.090522] Lustre: lustre-MDT0001: new disk, initializing [ 9515.119538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9515.130991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9515.133642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9516.393163] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9520.344763] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9522.334680] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9525.065252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9525.220847] Lustre: lustre-OST0000: new disk, initializing [ 9525.223391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9525.226087] Lustre: 474043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9525.255298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9527.455257] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9530.873068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9530.878730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9530.903417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9531.258121] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9532.356120] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing lnet_if_list [ 9534.259988] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9534.270353] LNet: Added LNI 192.168.206.115@tcp10 [8/256/0/180] [ 9536.401697] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9536.414924] LNet: Added LNI 192.168.206.115@tcp11 [8/256/0/180] [ 9538.035068] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9538.047171] LNet: Added LNI 192.168.206.115@tcp12 [8/256/0/180] [ 9539.612681] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9539.631980] LNet: Added LNI 192.168.206.115@tcp13 [8/256/0/180] [ 9541.107478] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9541.117087] LNet: Added LNI 192.168.206.115@tcp14 [8/256/0/180] [ 9542.625264] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9542.638897] LNet: Added LNI 192.168.206.115@tcp15 [8/256/0/180] [ 9544.467152] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9544.477128] LNet: Added LNI 192.168.206.115@tcp16 [8/256/0/180] [ 9546.405469] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9546.418828] LNet: Added LNI 192.168.206.115@tcp17 [8/256/0/180] [ 9548.261969] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9548.275309] LNet: Added LNI 192.168.206.115@tcp18 [8/256/0/180] [ 9550.142610] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9550.154434] LNet: Added LNI 192.168.206.115@tcp19 [8/256/0/180] [ 9551.817209] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9551.831513] LNet: Added LNI 192.168.206.115@tcp20 [8/256/0/180] [ 9553.554484] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9553.573777] LNet: Added LNI 192.168.206.115@tcp21 [8/256/0/180] [ 9555.694472] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9555.710265] LNet: Added LNI 192.168.206.115@tcp22 [8/256/0/180] [ 9557.718016] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9557.733913] LNet: Added LNI 192.168.206.115@tcp23 [8/256/0/180] [ 9559.609381] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9559.618430] LNet: Added LNI 192.168.206.115@tcp24 [8/256/0/180] [ 9561.666978] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9561.676834] LNet: Added LNI 192.168.206.115@tcp25 [8/256/0/180] [ 9563.761591] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9563.772358] LNet: Added LNI 192.168.206.115@tcp26 [8/256/0/180] [ 9565.616288] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9565.630415] LNet: Added LNI 192.168.206.115@tcp27 [8/256/0/180] [ 9567.603655] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9567.616710] LNet: Added LNI 192.168.206.115@tcp28 [8/256/0/180] [ 9569.910100] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9569.923587] LNet: Added LNI 192.168.206.115@tcp29 [8/256/0/180] [ 9571.997307] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9572.011348] LNet: Added LNI 192.168.206.115@tcp30 [8/256/0/180] [ 9574.265286] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9574.283383] LNet: Added LNI 192.168.206.115@tcp31 [8/256/0/180] [ 9576.803596] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9576.824602] LNet: Added LNI 192.168.206.115@tcp32 [8/256/0/180] [ 9579.037337] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9579.048208] LNet: Added LNI 192.168.206.115@tcp33 [8/256/0/180] [ 9581.341559] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9581.354325] LNet: Added LNI 192.168.206.115@tcp34 [8/256/0/180] [ 9583.479395] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9583.500333] LNet: Added LNI 192.168.206.115@tcp35 [8/256/0/180] [ 9586.261833] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9586.276579] LNet: Added LNI 192.168.206.115@tcp36 [8/256/0/180] [ 9588.680902] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9588.691272] LNet: Added LNI 192.168.206.115@tcp37 [8/256/0/180] [ 9590.705391] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9590.715965] LNet: Added LNI 192.168.206.115@tcp38 [8/256/0/180] [ 9593.152050] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9593.161363] LNet: Added LNI 192.168.206.115@tcp39 [8/256/0/180] [ 9595.585750] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9595.594413] LNet: Added LNI 192.168.206.115@tcp40 [8/256/0/180] [ 9597.956627] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9597.971455] LNet: Added LNI 192.168.206.115@tcp41 [8/256/0/180] [ 9600.624273] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9600.636494] LNet: Added LNI 192.168.206.115@tcp42 [8/256/0/180] [ 9603.876646] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9603.893320] LNet: Added LNI 192.168.206.115@tcp43 [8/256/0/180] [ 9607.124596] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9607.138049] LNet: Added LNI 192.168.206.115@tcp44 [8/256/0/180] [ 9610.408571] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9610.428323] LNet: Added LNI 192.168.206.115@tcp45 [8/256/0/180] [ 9614.274654] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9614.293687] LNet: Added LNI 192.168.206.115@tcp46 [8/256/0/180] [ 9617.919692] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9617.935560] LNet: Added LNI 192.168.206.115@tcp47 [8/256/0/180] [ 9622.164835] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9622.187129] LNet: Added LNI 192.168.206.115@tcp48 [8/256/0/180] [ 9626.237877] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9626.258673] LNet: Added LNI 192.168.206.115@tcp49 [8/256/0/180] [ 9629.765918] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9629.781675] LNet: Added LNI 192.168.206.115@tcp50 [8/256/0/180] [ 9643.491761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9643.494706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9643.496622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9643.504068] Lustre: Skipped 2 previous similar messages [ 9643.509836] Lustre: Skipped 1 previous similar message [ 9644.877902] Lustre: server umount lustre-MDT0000 complete [ 9646.601526] LustreError: 472871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191175 with bad export cookie 13092627301283442619 [ 9646.608617] LustreError: 472871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9646.610792] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9646.763299] Lustre: server umount lustre-MDT0001 complete [ 9655.908370] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9660.040228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.991310] Lustre: 471346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789191177/real 1789191177] req@00000000d3a6c247 x1876102772513152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789191193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9665.015864] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9668.127156] Lustre: 471346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789191180/real 1789191180] req@00000000ed78b05e x1876102772513280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789191196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9670.111173] Lustre: 471344:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789191182/real 1789191182] req@000000002a6a636f x1876102772513408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789191198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9672.223510] LustreError: 471343:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000022798467 x1876102772514304/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9672.432485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9673.439169] Lustre: 471345:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789191185/real 1789191185] req@00000000235c7595 x1876102772513664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789191201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9674.118727] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9677.535205] Lustre: 471346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789191190/real 1789191190] req@000000003ab6908f x1876102772513792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789191206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9678.055460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.223586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9679.956483] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9683.422286] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9683.441846] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9684.459450] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9686.000436] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9698.272375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9698.277465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9698.286403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9699.812163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9699.816271] Lustre: Skipped 2 previous similar messages [ 9701.481600] Lustre: server umount lustre-OST0000 complete [ 9703.904189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9703.906671] LustreError: Skipped 1 previous similar message [ 9703.908030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9703.911271] Lustre: Skipped 1 previous similar message [ 9703.912986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9709.425822] Lustre: server umount lustre-MDT0000 complete [ 9710.048287] LustreError: 498236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9710.053109] LustreError: 498236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9710.809316] LustreError: 472109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191239 with bad export cookie 13092627301283444677 [ 9710.811742] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9710.812695] LustreError: 472109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9710.933336] Lustre: server umount lustre-MDT0001 complete [ 9714.805603] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9715.928187] Key type lgssc unregistered [ 9716.086841] LNet: 501008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9716.090868] LNetError: 501008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9716.093894] LNet: Removed LNI 192.168.206.115@tcp [ 9716.095555] LNet: Removed LNI 192.168.206.115@tcp10 [ 9716.097365] LNet: Removed LNI 192.168.206.115@tcp11 [ 9716.098775] LNet: Removed LNI 192.168.206.115@tcp12 [ 9716.100609] LNet: Removed LNI 192.168.206.115@tcp13 [ 9716.102507] LNet: Removed LNI 192.168.206.115@tcp14 [ 9716.104394] LNet: Removed LNI 192.168.206.115@tcp15 [ 9716.106226] LNet: Removed LNI 192.168.206.115@tcp16 [ 9716.107909] LNet: Removed LNI 192.168.206.115@tcp17 [ 9716.109618] LNet: Removed LNI 192.168.206.115@tcp18 [ 9716.111503] LNet: Removed LNI 192.168.206.115@tcp19 [ 9716.113370] LNet: Removed LNI 192.168.206.115@tcp20 [ 9716.115086] LNet: Removed LNI 192.168.206.115@tcp21 [ 9716.116761] LNet: Removed LNI 192.168.206.115@tcp22 [ 9716.118496] LNet: Removed LNI 192.168.206.115@tcp23 [ 9716.120319] LNet: Removed LNI 192.168.206.115@tcp24 [ 9716.121956] LNet: Removed LNI 192.168.206.115@tcp25 [ 9716.123601] LNet: Removed LNI 192.168.206.115@tcp26 [ 9716.125341] LNet: Removed LNI 192.168.206.115@tcp27 [ 9716.126826] LNet: Removed LNI 192.168.206.115@tcp28 [ 9716.128349] LNet: Removed LNI 192.168.206.115@tcp29 [ 9716.129722] LNet: Removed LNI 192.168.206.115@tcp30 [ 9716.131290] LNet: Removed LNI 192.168.206.115@tcp31 [ 9716.132734] LNet: Removed LNI 192.168.206.115@tcp32 [ 9716.134284] LNet: Removed LNI 192.168.206.115@tcp33 [ 9716.135594] LNet: Removed LNI 192.168.206.115@tcp34 [ 9716.137149] LNet: Removed LNI 192.168.206.115@tcp35 [ 9716.138464] LNet: Removed LNI 192.168.206.115@tcp36 [ 9716.139937] LNet: Removed LNI 192.168.206.115@tcp37 [ 9716.141307] LNet: Removed LNI 192.168.206.115@tcp38 [ 9716.142576] LNet: Removed LNI 192.168.206.115@tcp39 [ 9716.143970] LNet: Removed LNI 192.168.206.115@tcp40 [ 9716.145317] LNet: Removed LNI 192.168.206.115@tcp41 [ 9716.146557] LNet: Removed LNI 192.168.206.115@tcp42 [ 9716.147828] LNet: Removed LNI 192.168.206.115@tcp43 [ 9716.149080] LNet: Removed LNI 192.168.206.115@tcp44 [ 9716.150259] LNet: Removed LNI 192.168.206.115@tcp45 [ 9716.151435] LNet: Removed LNI 192.168.206.115@tcp46 [ 9716.152587] LNet: Removed LNI 192.168.206.115@tcp47 [ 9716.153760] LNet: Removed LNI 192.168.206.115@tcp48 [ 9716.154992] LNet: Removed LNI 192.168.206.115@tcp49 [ 9716.160369] LNet: Removed LNI 192.168.206.115@tcp50 [ 9716.451706] Key type .llcrypt unregistered [ 9716.453239] Key type ._llcrypt unregistered [ 9725.231250] Key type ._llcrypt registered [ 9725.232944] Key type .llcrypt registered [ 9725.282877] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 9732.207187] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9732.509508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9732.549071] alg: No test for adler32 (adler32-zlib) [ 9733.394774] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 9733.473150] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9735.047170] Key type lgssc registered [ 9735.524736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9750.107013] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9750.115047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9751.215428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9751.228422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9751.277704] Lustre: lustre-MDT0000: new disk, initializing [ 9751.302092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9751.309573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9752.690263] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9758.008720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.088780] Lustre: 504174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9758.134398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9758.138186] Lustre: Skipped 1 previous similar message [ 9758.187423] Lustre: lustre-MDT0001: new disk, initializing [ 9758.222708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9758.238984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9758.244761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9759.765214] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9763.888760] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9765.907024] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9769.173931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9769.275654] Lustre: *** cfs_fail_loc=90f, val=0*** [ 9769.305112] Lustre: lustre-OST0000: new disk, initializing [ 9769.306797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9769.309111] Lustre: 506191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9769.342876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.637139] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9776.136940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9776.143991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9776.206764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9776.371571] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9781.222278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9781.223517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9781.227439] LustreError: Skipped 1 previous similar message [ 9781.237721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9781.242459] Lustre: Skipped 1 previous similar message [ 9785.595590] Lustre: server umount lustre-OST0000 complete [ 9788.897142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9788.901253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.906339] Lustre: Skipped 1 previous similar message [ 9788.909865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9791.457747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9791.459171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9793.732808] Lustre: server umount lustre-MDT0000 complete [ 9795.507693] LustreError: 504167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191324 with bad export cookie 3893660989576319676 [ 9795.515057] LustreError: 504167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9795.515837] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9795.714631] Lustre: server umount lustre-MDT0001 complete [ 9800.160586] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9801.808329] Key type lgssc unregistered [ 9801.995940] LNet: 508092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9802.001403] LNetError: 508092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9802.012789] LNet: Removed LNI 192.168.206.115@tcp [ 9802.512108] Key type .llcrypt unregistered [ 9802.514318] Key type ._llcrypt unregistered [ 9811.112923] Key type ._llcrypt registered [ 9811.113993] Key type .llcrypt registered [ 9811.154787] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9811.486718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9811.493943] alg: No test for adler32 (adler32-zlib) [ 9812.355691] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 9812.448076] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9814.031227] Key type lgssc registered [ 9814.379834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9817.452779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9817.461285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9818.623644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9820.294457] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9824.438825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.629232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9826.349232] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9829.205722] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9831.757164] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9834.174800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9834.327833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.076855] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9840.655703] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9850.535417] loop0: detected capacity change from 0 to 204800000 [ 9850.614219] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9850.617271] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9850.620333] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9853.165743] loop0: detected capacity change from 0 to 204800000 [ 9853.291967] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9853.297509] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9853.303204] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9855.482119] loop0: detected capacity change from 0 to 204800000 [ 9858.224053] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9858.309742] Lustre: fs15246-OST0000: new disk, initializing [ 9858.311594] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9858.314415] Lustre: 512945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9858.337436] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.892464] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9865.997549] loop1: detected capacity change from 0 to 204800000 [ 9868.625644] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9868.680376] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9868.718394] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9868.766318] Lustre: fs15246-MDT0000: new disk, initializing [ 9868.845638] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.860491] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9868.928666] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9869.984673] Lustre: 514067:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000002805df x1876103099614464/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789191414 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9870.790875] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9879.075691] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9879.081583] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9879.093706] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9882.092182] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9882.100791] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9887.206990] LustreError: 509347:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.213347] LustreError: 509347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9887.220440] Lustre: server umount fs15246-MDT0000 complete [ 9891.136228] Lustre: server umount fs15246-OST0000 complete [ 9896.928436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9896.933328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9896.940687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9901.792586] Lustre: server umount lustre-OST0000 complete [ 9906.656393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9906.661176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9906.668627] Lustre: Skipped 1 previous similar message [ 9906.672068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9906.676947] Lustre: Skipped 2 previous similar messages [ 9909.561945] Lustre: server umount lustre-MDT0000 complete [ 9911.316243] LustreError: 509332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191439 with bad export cookie 8740489673005995097 [ 9911.321635] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9911.322266] LustreError: 509332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9911.463820] Lustre: server umount lustre-MDT0001 complete [ 9915.872595] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9916.863251] Key type lgssc unregistered [ 9917.008420] LNet: 517159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9917.012395] LNetError: 517159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9917.022524] LNet: Removed LNI 192.168.206.115@tcp [ 9917.462599] Key type .llcrypt unregistered [ 9917.464404] Key type ._llcrypt unregistered [ 9926.655939] Key type ._llcrypt registered [ 9926.657015] Key type .llcrypt registered [ 9926.697572] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9927.017236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9927.069419] alg: No test for adler32 (adler32-zlib) [ 9927.915229] Lustre: Lustre: Build Version: 2.17.58_39_g3d58bdf [ 9927.996708] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9929.583230] Key type lgssc registered [ 9930.092848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9946.910305] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 9951.011937] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9964.169711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9964.179096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9965.273125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9965.284018] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9965.332717] Lustre: lustre-MDT0000: new disk, initializing [ 9965.354727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.361614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9966.849382] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9971.735508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.803067] Lustre: 521591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9971.840579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9971.844149] Lustre: Skipped 1 previous similar message [ 9971.891048] Lustre: lustre-MDT0001: new disk, initializing [ 9971.937181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9971.957222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9971.963200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9973.758582] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9978.425862] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9980.855608] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9983.887240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9983.982681] Lustre: lustre-OST0000: new disk, initializing [ 9983.984313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9983.986497] Lustre: 523559:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9984.018438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9986.254290] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9989.115996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9989.120074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9989.163842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9990.346372] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9991.895367] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9991.954907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9993.134162] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9993.194462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9994.210096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9994.212977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9994.217096] LustreError: Skipped 1 previous similar message [ 9994.226058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9994.233484] Lustre: Skipped 1 previous similar message [ 9996.130694] Lustre: server umount lustre-OST0000 complete [ 9999.328756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.329288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9999.331777] Lustre: Skipped 2 previous similar messages [ 9999.334298] Lustre: Skipped 1 previous similar message [10004.449106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10004.453034] Lustre: Skipped 1 previous similar message [10009.568991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10009.572832] Lustre: Skipped 1 previous similar message [10012.127128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10012.211363] Lustre: server umount lustre-MDT0000 complete [10013.624654] LustreError: 522385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191542 with bad export cookie 1953118191292634783 [10013.630038] LustreError: 522385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10013.630669] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10013.757556] Lustre: server umount lustre-MDT0001 complete [10023.492339] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [10027.947527] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10042.515967] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9795 sec ========== 01:39:29 (1789191569) [10043.284186] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:39:30 (1789191570) === [10044.816376] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:39:32 (1789191572) === [10060.180447] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [10061.359378] Key type lgssc unregistered [10061.523601] LNet: 530191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10061.528668] LNetError: 530191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10061.540573] LNet: Removed LNI 192.168.206.115@tcp [10061.988707] Key type .llcrypt unregistered [10061.990348] Key type ._llcrypt unregistered