[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 609156546 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002013] Switched APIC routing to physical x2apic. [ 0.003017] kvm-guest: setup PV IPIs [ 0.006899] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009162] LSM: Security Framework initializing [ 0.010056] Yama: becoming mindful. [ 0.011029] SELinux: Initializing. [ 0.012000] *** VALIDATE selinux *** [ 0.015651] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.019763] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.020000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.020000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.020000] *** VALIDATE tmpfs *** [ 0.021150] *** VALIDATE proc *** [ 0.023290] *** VALIDATE cgroup *** [ 0.024007] *** VALIDATE cgroup2 *** [ 0.025268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.027008] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.028006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.029029] Spectre V2 : User space: Vulnerable [ 0.030008] Speculative Store Bypass: Vulnerable [ 0.033556] debug: unmapping init [mem 0xffffffffa4e59000-0xffffffffa4e60fff] [ 0.036168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.037648] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.038019] ... version: 2 [ 0.039011] ... bit width: 48 [ 0.040009] ... generic registers: 4 [ 0.041010] ... value mask: 0000ffffffffffff [ 0.042012] ... max period: 00007fffffffffff [ 0.043012] ... fixed-purpose events: 3 [ 0.044009] ... event mask: 000000070000000f [ 0.046354] rcu: Hierarchical SRCU implementation. [ 0.048796] smp: Bringing up secondary CPUs ... [ 0.049633] x86: Booting SMP configuration: [ 0.050028] .... node #0, CPUs: #1 #2 #3 [ 0.059016] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107273] node 0 deferred pages initialised in 43ms [ 0.111138] devtmpfs: initialized [ 0.113285] x86/mm: Memory block size: 128MB [ 0.116202] gcov: version magic: 0x41383552 [ 0.118329] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123425] pinctrl core: initialized pinctrl subsystem [ 0.125168] [ 0.125863] ************************************************************* [ 0.129015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131011] ** ** [ 0.134014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.136011] ** ** [ 0.139011] ** This means that this kernel is built to expose internal ** [ 0.141011] ** IOMMU data structures, which may compromise security on ** [ 0.144011] ** your system. ** [ 0.146011] ** ** [ 0.149013] ** If you see this message and you are not debugging the ** [ 0.151013] ** kernel, report this immediately to your vendor! ** [ 0.154011] ** ** [ 0.157010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159011] ************************************************************* [ 0.163274] NET: Registered protocol family 16 [ 0.164552] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171146] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175103] cpuidle: using governor menu [ 0.176848] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177714] PCI: Using configuration type 1 for base access [ 0.179149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195028] cryptd: max_cpu_qlen set to 1000 [ 0.198302] ACPI: Added _OSI(Module Device) [ 0.200016] ACPI: Added _OSI(Processor Device) [ 0.202012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209724] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220486] ACPI: Interpreter enabled [ 0.224093] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227016] ACPI: Using IOAPIC for interrupt routing [ 0.230815] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239541] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257349] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269127] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274703] acpiphp: Slot [2] registered [ 0.276144] acpiphp: Slot [5] registered [ 0.278213] acpiphp: Slot [6] registered [ 0.279116] acpiphp: Slot [7] registered [ 0.281139] acpiphp: Slot [8] registered [ 0.282244] acpiphp: Slot [9] registered [ 0.284277] acpiphp: Slot [10] registered [ 0.285256] acpiphp: Slot [3] registered [ 0.287112] acpiphp: Slot [4] registered [ 0.289118] acpiphp: Slot [11] registered [ 0.290116] acpiphp: Slot [12] registered [ 0.292097] acpiphp: Slot [13] registered [ 0.293099] acpiphp: Slot [14] registered [ 0.295151] acpiphp: Slot [15] registered [ 0.297092] acpiphp: Slot [16] registered [ 0.298203] acpiphp: Slot [17] registered [ 0.300142] acpiphp: Slot [18] registered [ 0.302103] acpiphp: Slot [19] registered [ 0.303151] acpiphp: Slot [20] registered [ 0.305131] acpiphp: Slot [21] registered [ 0.307083] acpiphp: Slot [22] registered [ 0.308085] acpiphp: Slot [23] registered [ 0.310081] acpiphp: Slot [24] registered [ 0.311082] acpiphp: Slot [25] registered [ 0.313091] acpiphp: Slot [26] registered [ 0.314192] acpiphp: Slot [27] registered [ 0.316089] acpiphp: Slot [28] registered [ 0.318080] acpiphp: Slot [29] registered [ 0.319087] acpiphp: Slot [30] registered [ 0.321078] acpiphp: Slot [31] registered [ 0.322054] PCI host bridge to bus 0000:00 [ 0.324017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338296] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342163] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345251] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354835] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.360055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.371698] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373948] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379962] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.406025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.411962] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.420016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.442022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455328] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469125] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.478018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.496021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.509251] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.516015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.542023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.548074] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.551000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.552000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.565015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.570000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.575029] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.579021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.599022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.610333] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.617016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.622021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.636017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.647875] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.648357] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.651398] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.653281] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.655222] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.660300] iommu: Default domain type: Passthrough [ 0.661000] SCSI subsystem initialized [ 0.661000] ACPI: bus type USB registered [ 0.663133] usbcore: registered new interface driver usbfs [ 0.665069] usbcore: registered new interface driver hub [ 0.667092] usbcore: registered new device driver usb [ 0.668214] pps_core: LinuxPPS API ver. 1 registered [ 0.670011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.672149] PTP clock support registered [ 0.674401] EDAC MC: Ver: 3.0.0 [ 0.676124] PCI: Using ACPI for IRQ routing [ 0.678902] NetLabel: Initializing [ 0.680017] NetLabel: domain hash size = 128 [ 0.681010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.683143] NetLabel: unlabeled traffic allowed by default [ 0.685336] vgaarb: loaded [ 0.687614] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.689007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.694334] clocksource: Switched to clocksource kvm-clock [ 0.815616] VFS: Disk quotas dquot_6.6.0 [ 0.817072] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.819271] *** VALIDATE ramfs *** [ 0.820338] *** VALIDATE hugetlbfs *** [ 0.821810] pnp: PnP ACPI init [ 0.824150] pnp: PnP ACPI: found 6 devices [ 0.839688] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.842896] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.844744] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.846469] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.848512] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.850434] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.852949] NET: Registered protocol family 2 [ 0.855061] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.859654] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.862388] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.867558] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.870580] TCP: Hash tables configured (established 65536 bind 65536) [ 0.873147] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.875760] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.878086] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.881178] NET: Registered protocol family 1 [ 0.883352] RPC: Registered named UNIX socket transport module. [ 0.885326] RPC: Registered udp transport module. [ 0.886779] RPC: Registered tcp transport module. [ 0.888253] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.890259] NET: Registered protocol family 44 [ 0.891625] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.893256] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.894967] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.896872] PCI: CLS 0 bytes, default 64 [ 0.898787] Unpacking initramfs... [ 2.669900] debug: unmapping init [mem 0xffff88cefcc54000-0xffff88cefffbffff] [ 2.676517] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.679029] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.683341] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.333174] Initialise system trusted keyrings [ 3.334203] Key type blacklist registered [ 3.338503] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.347864] zbud: loaded [ 3.351057] *** VALIDATE nfs *** [ 3.352243] *** VALIDATE nfs4 *** [ 3.353965] pstore: using deflate compression [ 3.357969] Platform Keyring initialized [ 3.492595] NET: Registered protocol family 38 [ 3.500380] Key type asymmetric registered [ 3.501966] Asymmetric key parser 'x509' registered [ 3.504184] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.507315] io scheduler mq-deadline registered [ 3.508844] io scheduler kyber registered [ 3.510703] io scheduler bfq registered [ 3.512650] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.516205] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.519591] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.522697] ACPI: Power Button [PWRF] [ 3.528681] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.536306] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.551282] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.562678] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.577461] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.607073] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.637420] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.642104] Non-volatile memory driver v1.3 [ 3.643758] Linux agpgart interface v0.103 [ 3.674040] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.676860] vda: detected capacity change from 0 to 75087872 [ 3.689886] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.692822] vdb: detected capacity change from 0 to 1073741824 [ 3.709257] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.712431] vdc: detected capacity change from 0 to 2621440000 [ 3.726994] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.730319] vdd: detected capacity change from 0 to 2621440000 [ 3.744493] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.747572] vde: detected capacity change from 0 to 4294967296 [ 3.761320] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.764255] vdf: detected capacity change from 0 to 4294967296 [ 3.770699] libphy: Fixed MDIO Bus: probed [ 3.783318] usbcore: registered new interface driver usbserial_generic [ 3.785703] usbserial: USB Serial support registered for generic [ 3.787934] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.792554] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.794444] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.796989] mousedev: PS/2 mouse device common for all mice [ 3.799864] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.802348] rtc_cmos 00:05: RTC can wake from S4 [ 3.808682] rtc_cmos 00:05: registered as rtc0 [ 3.810659] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.813820] intel_pstate: CPU model not supported [ 3.815302] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.823834] hid: raw HID events driver (C) Jiri Kosina [ 3.825939] usbcore: registered new interface driver usbhid [ 3.828669] usbhid: USB HID core driver [ 3.830416] drop_monitor: Initializing network drop monitor service [ 3.832778] Initializing XFRM netlink socket [ 3.834947] NET: Registered protocol family 10 [ 3.835820] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.844175] Segment Routing with IPv6 [ 3.845651] NET: Registered protocol family 17 [ 3.847566] mpls_gso: MPLS GSO support [ 3.852823] RAS: Correctable Errors collector initialized. [ 3.854986] AVX version of gcm_enc/dec engaged. [ 3.856675] AES CTR mode by8 optimization enabled [ 3.939522] sched_clock: Marking stable (3939504997, 0)->(5086544246, -1147039249) [ 3.942863] registered taskstats version 1 [ 3.945780] Loading compiled-in X.509 certificates [ 3.947862] zswap: loaded using pool lzo/zbud [ 3.973541] Key type big_key registered [ 3.987177] Key type encrypted registered [ 3.989227] ima: No TPM chip found, activating TPM-bypass! [ 3.991536] ima: Allocated hash algorithm: sha1 [ 4.001309] ima: No architecture policies found [ 4.003242] evm: Initialising EVM extended attributes: [ 4.005499] evm: security.selinux [ 4.006837] evm: security.ima [ 4.008178] evm: security.capability [ 4.009620] evm: HMAC attrs: 0x1 [ 4.012253] rtc_cmos 00:05: setting system clock to 2026-09-13 01:39:29 UTC (1789263569) [ 4.019217] debug: unmapping init [mem 0xffffffffa5e03000-0xffffffffa5ffffff] [ 4.022625] debug: unmapping init [mem 0xffffffffa4b82000-0xffffffffa4e58fff] [ 4.030183] Write protecting the kernel read-only data: 28672k [ 4.033799] debug: unmapping init [mem 0xffffffffa3203000-0xffffffffa33fffff] [ 4.036768] debug: unmapping init [mem 0xffffffffa3b14000-0xffffffffa3bfffff] [ 4.070684] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.080566] systemd[1]: Detected virtualization kvm. [ 4.082479] systemd[1]: Detected architecture x86-64. [ 4.084345] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.111607] systemd[1]: No hostname configured. [ 4.113149] systemd[1]: Set hostname to . [ 4.114819] random: systemd: uninitialized urandom read (16 bytes read) [ 4.116810] systemd[1]: Initializing machine ID from random generator. [ 4.271373] random: systemd: uninitialized urandom read (16 bytes read) [ 4.273610] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.277489] random: systemd: uninitialized urandom read (16 bytes read) [ 4.279798] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.284043] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.994615] device-mapper: uevent: version 1.0.3 [ 4.996944] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Mounting Kernel Configuration File System... [ 5.959102] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Mounted Kernel Configuration File System. [ 6.126623] scsi host0: ata_piix [ 6.203140] random: fast init done [ 6.279748] scsi host1: ata_piix [ 6.281515] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.284118] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.516809] random: crng init done [ 10.521168] random: 7 urandom warning(s) missed due to ratelimiting [ 11.741249] dracut-initqueue[575]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached targ[ 12.765282] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) et Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.387837] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.864945] SELinux: Disabled at runtime. [ 14.936143] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.945359] systemd[1]: Detected virtualization kvm. [ 14.947402] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.854833] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.858858] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.869127] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.875615] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.879942] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.890888] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.898745] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ 15.955962] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 17.819768] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.561224] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.630359] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.189013] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.267957] EDAC sbridge: Ver: 1.1.2 [ 22.711377] Key type dns_resolver registered [ 23.152976] NFS: Registering the id_resolver key type [ 23.155352] Key type id_resolver registered [ 23.181815] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg251-server login: [ 66.456538] libcfs: loading out-of-tree module taints kernel. [ 66.480168] Key type ._llcrypt registered [ 66.481668] Key type .llcrypt registered [ 66.540511] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 75.059609] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 75.699342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.706353] alg: No test for adler32 (adler32-zlib) [ 76.738581] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 77.131115] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 78.759214] Key type lgssc registered [ 79.583734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.239644] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 106.778425] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 112.367666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 112.382192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 113.523431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 113.536825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 113.588466] Lustre: lustre-MDT0000: new disk, initializing [ 113.623360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.633383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 115.403313] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.929423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 121.985171] Lustre: 6464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 122.004814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 122.007548] Lustre: Skipped 1 previous similar message [ 122.051375] Lustre: lustre-MDT0001: new disk, initializing [ 122.093839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 122.111048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 122.121840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 123.946233] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 126.625575] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 130.472416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 130.575559] Lustre: lustre-OST0000: new disk, initializing [ 130.577522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 130.580322] Lustre: 8398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 130.602980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 133.068436] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.161163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 135.171066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 135.216558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 140.107068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 140.171570] Lustre: lustre-OST0001: new disk, initializing [ 140.173421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 140.175933] Lustre: 9471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 140.213077] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 141.617863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 141.624790] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 141.653162] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 142.603128] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.404709] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 153.767232] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 156.694814] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing check_logdir /tmp/testlogs/ [ 159.113640] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing yml_node [ 161.155223] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 162.348529] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 163.656959] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 164.494490] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 12 21:42:09 EDT 2026 [ 172.914981] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 173.632151] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 187.873435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 187.874453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 187.883596] Lustre: Skipped 3 previous similar messages [ 187.890229] Lustre: Skipped 2 previous similar messages [ 192.366598] Lustre: server umount lustre-MDT0000 complete [ 194.404187] LustreError: 6456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789263759 with bad export cookie 5290797744748151041 [ 194.404319] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 194.409450] LustreError: 6456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 198.111639] LustreError: 6472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 198.111731] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 198.112039] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 198.119966] LustreError: 6472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 198.132052] Lustre: Skipped 1 previous similar message [ 199.135518] LustreError: 6470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 199.135943] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 199.141439] LustreError: 6470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 199.144331] Lustre: Skipped 2 previous similar messages [ 200.647475] Lustre: server umount lustre-MDT0001 complete [ 202.924157] Lustre: server umount lustre-OST0000 complete [ 205.166314] Lustre: server umount lustre-OST0001 complete [ 208.509363] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 213.858665] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 225.434066] hrtimer: interrupt took 8703911 ns [ 234.246534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 234.406674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 234.426635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 234.478514] Lustre: lustre-MDT0000: new disk, initializing [ 234.538062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.547945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 237.290205] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 244.409484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.498569] Lustre: 15012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 244.542457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 244.545147] Lustre: Skipped 1 previous similar message [ 244.598447] Lustre: lustre-MDT0001: new disk, initializing [ 244.672158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.707303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 244.715457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 247.273210] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 253.731773] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 257.163648] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 261.282025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.449457] Lustre: lustre-OST0000: new disk, initializing [ 261.453589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 261.457908] Lustre: 16978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.523259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.939926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 262.945580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 262.997706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 265.362293] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 271.781708] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 274.400799] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 274.535852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 277.198791] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 277.331116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 278.497049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 278.498921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 278.504646] Lustre: Skipped 1 previous similar message [ 278.505527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 278.508909] LustreError: Skipped 1 previous similar message [ 278.520443] Lustre: Skipped 1 previous similar message [ 283.622432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 283.628413] Lustre: Skipped 1 previous similar message [ 284.605660] Lustre: server umount lustre-OST0000 complete [ 288.743261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.750595] Lustre: Skipped 1 previous similar message [ 293.071966] Lustre: server umount lustre-MDT0000 complete [ 293.861109] LustreError: 15023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.871935] LustreError: 15023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 295.372691] LustreError: 15817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789263860 with bad export cookie 5290797744748152574 [ 295.375590] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.379789] LustreError: 15817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 295.597342] Lustre: server umount lustre-MDT0001 complete [ 298.303415] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:44:22 (1789263862) [ 299.757166] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 300.828729] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:44:25 (1789263865) [ 304.526607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 304.797355] LustreError: 19385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.847284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 307.258983] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 310.240641] LustreError: 19386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.023923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 314.682386] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 318.972483] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 321.987372] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 325.703736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 328.960397] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 333.047382] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 340.448239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 340.451737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.459031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 340.461672] Lustre: Skipped 2 previous similar messages [ 342.498974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 346.505619] Lustre: server umount lustre-OST0000 complete [ 352.736980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.753765] Lustre: Skipped 2 previous similar messages [ 356.148966] Lustre: server umount lustre-MDT0000 complete [ 357.880560] LustreError: 21223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.892884] LustreError: 21223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 360.112794] LustreError: 19366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789263925 with bad export cookie 5290797744748153456 [ 360.115691] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.120664] LustreError: 19366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 360.634217] Lustre: server umount lustre-MDT0001 complete [ 370.997196] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 374.807537] Key type lgssc unregistered [ 375.120868] LNet: 22918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.125574] LNetError: 22918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 375.151157] LNet: Removed LNI 192.168.202.151@tcp [ 376.906154] Key type .llcrypt unregistered [ 376.908062] Key type ._llcrypt unregistered [ 404.273792] Key type ._llcrypt registered [ 404.277938] Key type .llcrypt registered [ 404.431273] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 405.612220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 405.641510] alg: No test for adler32 (adler32-zlib) [ 406.755494] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 407.080856] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 408.719351] Key type lgssc registered [ 409.798866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 417.899340] loop: module loaded [ 417.984130] loop0: detected capacity change from 0 to 204800000 [ 418.248262] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 418.269888] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 418.557685] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 424.743986] loop0: detected capacity change from 0 to 204800000 [ 424.959494] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 424.970182] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 424.998571] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 429.874175] loop0: detected capacity change from 0 to 204800000 [ 436.325982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 436.345921] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.592366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 437.646923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 437.760414] Lustre: lustre-MDT0000: new disk, initializing [ 437.844996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.861951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 441.434695] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 449.761043] loop1: detected capacity change from 0 to 204800000 [ 454.442257] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 454.747403] Lustre: lustre-OST07c6: new disk, initializing [ 454.757122] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 454.762873] Lustre: Skipped 1 previous similar message [ 454.771098] Lustre: 26101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 454.890674] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 461.136623] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 461.433388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 461.457208] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 461.505572] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 466.407115] Lustre: 25087:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 466.413694] Lustre: 25087:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 466.418832] Lustre: 25087:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 466.424783] Lustre: 25087:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 466.430293] Lustre: 25087:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 466.434376] Lustre: 25087:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 482.701391] Lustre: 25088:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 482.707229] Lustre: 25088:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 482.711784] Lustre: 25088:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 482.715449] Lustre: 25088:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 482.719737] Lustre: 25088:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 482.723462] Lustre: 25088:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 486.998246] Lustre: 25088:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 487.004379] Lustre: 25088:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 487.015953] Lustre: 25088:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 487.025837] Lustre: 25088:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.043029] Lustre: 25088:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 487.062514] Lustre: 25088:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.067547] Lustre: 25088:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 487.072943] Lustre: 25088:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.084228] Lustre: 25088:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 487.090355] Lustre: 25088:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.096835] Lustre: 25088:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 487.103150] Lustre: 25088:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 490.272616] Lustre: 26109:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 490.289172] Lustre: 26109:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 490.297039] Lustre: 26109:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 490.304246] Lustre: 26109:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 490.310521] Lustre: 26109:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 490.315474] Lustre: 26109:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 490.323354] Lustre: 26109:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 490.328370] Lustre: 26109:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 490.335076] Lustre: 26109:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 490.343829] Lustre: 26109:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 490.347770] Lustre: 26109:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 490.354992] Lustre: 26109:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 494.642918] Lustre: 26109:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 494.649490] Lustre: 26109:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 494.656380] Lustre: 26109:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 494.661834] Lustre: 26109:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 494.668132] Lustre: 26109:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 494.673744] Lustre: 26109:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 494.679754] Lustre: 26109:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 494.691752] Lustre: 26109:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 494.703843] Lustre: 26109:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 494.709203] Lustre: 26109:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 494.716310] Lustre: 26109:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 494.727408] Lustre: 26109:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 505.240090] Lustre: 25089:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 505.244658] Lustre: 25089:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 505.252362] Lustre: 25089:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 505.262226] Lustre: 25089:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 505.270813] Lustre: 25089:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 505.281064] Lustre: 25089:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 505.285160] Lustre: 25089:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 505.292552] Lustre: 25089:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 505.299594] Lustre: 25089:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 505.304685] Lustre: 25089:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 505.309591] Lustre: 25089:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 505.314565] Lustre: 25089:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 553.441939] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.453525] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.464494] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 556.750958] Lustre: server umount lustre-OST07c6 complete [ 564.368485] Lustre: server umount lustre-MDT0000 complete [ 576.588460] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 578.612050] Key type lgssc unregistered [ 578.869666] LNet: 28655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.872709] LNetError: 28655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 578.891672] LNet: Removed LNI 192.168.202.151@tcp [ 579.505633] Key type .llcrypt unregistered [ 579.508483] Key type ._llcrypt unregistered [ 593.885721] Key type ._llcrypt registered [ 593.887745] Key type .llcrypt registered [ 593.989174] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 594.678062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 594.745251] alg: No test for adler32 (adler32-zlib) [ 595.760481] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 595.962255] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 597.623189] Key type lgssc registered [ 598.446789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 604.329751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 604.349531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.749875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.512251] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 615.907427] LustreError: 29900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.924321] LustreError: 29900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 616.419181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.706161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 619.918557] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 625.284187] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 629.102464] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 633.757288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 634.064258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.806893] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 644.152617] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 651.501093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 664.038876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 664.043532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.051585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 667.117734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.123115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 667.137786] Lustre: Skipped 1 previous similar message [ 672.227052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 672.239801] Lustre: Skipped 1 previous similar message [ 677.348847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 677.356329] Lustre: Skipped 1 previous similar message [ 678.367167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 678.521538] Lustre: server umount lustre-OST0000 complete [ 682.466717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.469588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 682.478295] Lustre: Skipped 1 previous similar message [ 682.499184] Lustre: Skipped 1 previous similar message [ 687.871550] Lustre: server umount lustre-MDT0000 complete [ 691.159324] LustreError: 29880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264256 with bad export cookie 14424253876026707449 [ 691.159324] LustreError: 29882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264256 with bad export cookie 14424253876026707449 [ 691.159648] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 691.389403] Lustre: server umount lustre-MDT0001 complete [ 699.285432] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 701.428313] Key type lgssc unregistered [ 701.648350] LNet: 33657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.653309] LNetError: 33657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.668278] LNet: Removed LNI 192.168.202.151@tcp [ 702.239280] Key type .llcrypt unregistered [ 702.241577] Key type ._llcrypt unregistered [ 715.366183] Key type ._llcrypt registered [ 715.367735] Key type .llcrypt registered [ 715.446372] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 716.126040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 716.182668] alg: No test for adler32 (adler32-zlib) [ 717.165077] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 717.403401] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 719.071243] Key type lgssc registered [ 719.753468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 725.231340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 725.245978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.609568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.677875] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 735.579516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.810319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 739.000611] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 743.853242] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 747.864420] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 753.212762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 753.563937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.097452] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 762.935473] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 772.023519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 781.281956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 781.285218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.298963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 782.304670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 782.309655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 787.249871] Lustre: server umount lustre-OST0000 complete [ 792.035868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 792.040944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.054622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 792.058072] Lustre: Skipped 1 previous similar message [ 796.523807] Lustre: server umount lustre-MDT0000 complete [ 797.667076] LustreError: 34903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 797.685217] LustreError: 34903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 799.362318] LustreError: 35636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264364 with bad export cookie 4646493319607830473 [ 799.365191] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 799.371934] LustreError: 35636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 799.723913] Lustre: server umount lustre-MDT0001 complete [ 808.176705] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 810.875733] Key type lgssc unregistered [ 811.079519] LNet: 38455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.083314] LNetError: 38455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.099466] LNet: Removed LNI 192.168.202.151@tcp [ 811.901889] Key type .llcrypt unregistered [ 811.904289] Key type ._llcrypt unregistered [ 827.983455] Key type ._llcrypt registered [ 827.987242] Key type .llcrypt registered [ 828.101448] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 828.812119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 828.824753] alg: No test for adler32 (adler32-zlib) [ 829.838735] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 830.022666] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 831.647427] Key type lgssc registered [ 832.557721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.689712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.722659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.119092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.397136] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 849.705880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.950700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 853.369851] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 858.473056] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 862.537803] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 867.454701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 867.712323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 871.585713] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 876.182818] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 886.126405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 893.178662] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 894.207416] Lustre: server umount lustre-MDT0000 complete [ 895.969669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 895.984167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.009979] LustreError: 39847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 896.021787] LustreError: 39847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 896.484071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.558491] LustreError: 42445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264462 with bad export cookie 8030144084836526305 [ 896.566320] LustreError: 42445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 896.569869] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 898.293213] LustreError: 42637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.302501] LustreError: 42637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 898.313258] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 898.853789] Lustre: server umount lustre-MDT0001 complete [ 910.448378] LustreError: 42445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.51@tcp arrived at 1789264475 with bad export cookie 8030144084836526970 [ 910.462118] LustreError: 42445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 928.637616] Lustre: server umount lustre-OST0000 complete [ 938.576966] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 940.166495] Key type lgssc unregistered [ 940.345465] LNet: 43727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.349726] LNetError: 43727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.357421] LNet: Removed LNI 192.168.202.151@tcp [ 940.789508] Key type .llcrypt unregistered [ 940.790928] Key type ._llcrypt unregistered [ 954.065144] Key type ._llcrypt registered [ 954.066422] Key type .llcrypt registered [ 954.121884] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 954.520532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 954.553812] alg: No test for adler32 (adler32-zlib) [ 955.497278] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 955.611593] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 957.239169] Key type lgssc registered [ 957.820875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 961.473132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 961.483583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.706442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.844558] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 969.441499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.638577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 971.988953] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 976.413530] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 980.505542] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 984.987280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.185717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.275256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 988.948507] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 994.178857] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1000.929852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.932357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1000.943916] Lustre: Skipped 1 previous similar message [ 1001.719337] Lustre: lustre-OST0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 1001.723830] Lustre: Skipped 1 previous similar message [ 1005.572359] Lustre: server umount lustre-OST0000 complete [ 1020.897307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1020.905351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.921727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.913840] Lustre: server umount lustre-MDT0000 complete [ 1027.554188] LustreError: 45140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.569393] LustreError: 45140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1029.243151] LustreError: 45103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264594 with bad export cookie 11177468108810616812 [ 1029.250461] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.508927] Lustre: server umount lustre-MDT0001 complete [ 1041.822908] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1043.516518] Key type lgssc unregistered [ 1043.763071] LNet: 48822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.776073] LNetError: 48822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.799708] LNet: Removed LNI 192.168.202.151@tcp [ 1044.361218] Key type .llcrypt unregistered [ 1044.363149] Key type ._llcrypt unregistered [ 1058.909847] Key type ._llcrypt registered [ 1058.912325] Key type .llcrypt registered [ 1058.979715] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1059.505881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.625192] alg: No test for adler32 (adler32-zlib) [ 1060.634741] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1060.812376] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1062.455392] Key type lgssc registered [ 1063.378186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.447776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.467395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.825493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.640181] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1078.117882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.414405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1081.181373] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1086.095183] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1089.365800] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1093.781595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1094.086335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.094326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1097.818173] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1102.779632] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1106.887443] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1109.178308] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1120.373164] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1221h58m04s [ 1121.185886] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1123.809503] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.811822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.817330] Lustre: Skipped 2 previous similar messages [ 1123.820405] Lustre: Skipped 1 previous similar message [ 1124.593734] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 1124.599182] Lustre: Skipped 1 previous similar message [ 1125.675358] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 1125.682485] Lustre: Skipped 3 previous similar messages [ 1128.433673] Lustre: server umount lustre-MDT0000 complete [ 1130.501866] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1130.976809] LustreError: 50214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1130.988971] LustreError: 50214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1131.753096] LustreError: 50211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.525898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.613339] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1134.775632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.810476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1135.794681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1137.400782] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1139.525415] Lustre: DEBUG MARKER: Wait for df (36626) ... [ 1140.555124] Lustre: DEBUG MARKER: done [ 1144.287996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1144.293529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.303404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.312100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.319756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.883646] Lustre: server umount lustre-OST0000 complete [ 1156.577041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.578493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1156.580429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.580437] Lustre: Skipped 3 previous similar messages [ 1156.581471] Lustre: Skipped 1 previous similar message [ 1157.716599] Lustre: server umount lustre-MDT0000 complete [ 1160.048615] LustreError: 50196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264725 with bad export cookie 12124514817996560659 [ 1160.050936] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.054930] LustreError: 50196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1160.242645] Lustre: server umount lustre-MDT0001 complete [ 1166.238246] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1167.832521] Key type lgssc unregistered [ 1168.058802] LNet: 55125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.063681] LNetError: 55125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.124672] LNet: Removed LNI 192.168.202.151@tcp [ 1169.706812] Key type .llcrypt unregistered [ 1169.707930] Key type ._llcrypt unregistered [ 1191.003213] Key type ._llcrypt registered [ 1191.006719] Key type .llcrypt registered [ 1191.060426] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1191.574230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1191.599274] alg: No test for adler32 (adler32-zlib) [ 1192.527559] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1192.677334] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1194.311820] Key type lgssc registered [ 1194.949920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1199.568928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1199.580662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.750457] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1200.761501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1200.862232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.279680] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1208.642776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.713108] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1208.729785] Lustre: 57337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1208.896734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.851617] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1216.308480] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1219.720677] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1224.245360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1224.420230] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1224.577317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.679477] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1232.516418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1235.597910] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1241.673679] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1222h00m05s [ 1242.757964] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1247.298515] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1248.915307] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1248.974843] LustreError: 57345:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1269.501850] Lustre: lustre-MDT0000: Client 588410a0-5367-4ff0-bf87-077f80d10fb6 (at 192.168.202.51@tcp) reconnecting [ 1289.039213] LustreError: 57345:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1290.335751] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1291.704561] Lustre: DEBUG MARKER: done [ 1294.309144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.309281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1294.310342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1294.315927] Lustre: Skipped 1 previous similar message [ 1294.319038] LustreError: Skipped 1 previous similar message [ 1299.425609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1299.429148] Lustre: Skipped 2 previous similar messages [ 1300.183528] Lustre: server umount lustre-OST0000 complete [ 1304.545250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.547499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1304.558933] Lustre: Skipped 1 previous similar message [ 1304.570211] Lustre: Skipped 1 previous similar message [ 1308.466230] Lustre: server umount lustre-MDT0000 complete [ 1310.741647] LustreError: 60099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264876 with bad export cookie 7157866559702368590 [ 1310.742458] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1310.748557] LustreError: 60099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1310.916444] Lustre: server umount lustre-MDT0001 complete [ 1316.757639] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1318.387909] Key type lgssc unregistered [ 1318.573791] LNet: 61676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.579950] LNetError: 61676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1318.596532] LNet: Removed LNI 192.168.202.151@tcp [ 1319.143850] Key type .llcrypt unregistered [ 1319.145764] Key type ._llcrypt unregistered [ 1346.730499] Key type ._llcrypt registered [ 1346.732206] Key type .llcrypt registered [ 1346.819113] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1347.578967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1347.732080] alg: No test for adler32 (adler32-zlib) [ 1348.685788] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1348.819944] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1350.423175] Key type lgssc registered [ 1351.152123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1353.078653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1353.088517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.267716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1354.283315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1354.406048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.060348] Lustre: Failing over lustre-MDT0000 [ 1355.268778] Lustre: server umount lustre-MDT0000 complete [ 1359.528281] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:02:03 (1789264923) [ 1363.888761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.106678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.843877] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1372.254325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.319756] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1372.335439] Lustre: 64447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1372.518690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1375.837601] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1381.500799] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1388.694449] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1399.588741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.727479] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1399.896730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.767358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1410.177954] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1419.556609] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1426.964995] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1435.620724] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.628885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1435.651249] Lustre: Skipped 2 previous similar messages [ 1435.658835] Lustre: Skipped 1 previous similar message [ 1440.741962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1440.756790] Lustre: Skipped 3 previous similar messages [ 1441.952286] Lustre: server umount lustre-MDT0000 complete [ 1445.707311] LustreError: 64440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265011 with bad export cookie 5340977819410007827 [ 1445.711222] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1445.720622] LustreError: 64440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1445.857280] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.891748] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1446.228735] Lustre: server umount lustre-MDT0001 complete [ 1450.044242] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1456.007502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1467.490876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.994171] LustreError: 66296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1472.024746] LustreError: 66296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1472.170566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.380849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1476.356614] LustreError: 66297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1476.382173] LustreError: 66297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1476.393358] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1477.531770] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1481.702152] LustreError: 66296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.795480] LustreError: 66297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.816958] LustreError: 66297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1489.406554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.923202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1495.041562] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1495.885327] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1504.870870] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1512.581835] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1520.609989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1520.617634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.627650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.632641] Lustre: Skipped 1 previous similar message [ 1520.644915] Lustre: Skipped 2 previous similar messages [ 1525.730043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1525.750811] Lustre: Skipped 2 previous similar messages [ 1526.807478] Lustre: server umount lustre-MDT0000 complete [ 1530.868240] LustreError: 66297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.873619] LustreError: 64441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265096 with bad export cookie 5340977819410010109 [ 1530.883764] LustreError: 66297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1530.893354] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.902871] LustreError: 64441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1531.275876] Lustre: server umount lustre-MDT0001 complete [ 1539.023410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1548.896600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.351440] Lustre: 63371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789265101/real 1789265101] req@00000000a1bf3fc9 x1876180246534016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789265117 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1552.396194] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.411473] Lustre: Skipped 1 previous similar message [ 1556.976370] LustreError: 66296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1557.149936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.231535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1558.049356] Lustre: 63372:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789265106/real 1789265106] req@00000000dfeb953a x1876180246534144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789265122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1561.267392] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1562.124084] Lustre: 63373:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789265111/real 1789265111] req@00000000075d57fc x1876180246534400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789265127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1562.141483] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1568.213509] Lustre: 63373:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789265116/real 1789265116] req@00000000c7133f35 x1876180246534528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789265132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1572.215252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.973550] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1578.164689] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1588.036550] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1594.850492] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1602.981513] Lustre: server umount lustre-MDT0000 complete [ 1603.555633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1603.561099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.573078] LustreError: 66297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.573089] LustreError: 66297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 1603.688564] Lustre: Skipped 1 previous similar message [ 1606.946822] LustreError: 65182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265172 with bad export cookie 5340977819410011488 [ 1606.949727] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1606.956288] LustreError: 65182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1607.189458] Lustre: server umount lustre-MDT0001 complete [ 1613.340991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1616.546590] Lustre: DEBUG MARKER: files compared the same [ 1628.761627] Lustre: server umount lustre-OST0000 complete [ 1651.292644] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1654.545586] Key type lgssc unregistered [ 1654.914806] LNet: 74191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.923191] LNetError: 74191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.941911] LNet: Removed LNI 192.168.202.151@tcp [ 1655.968415] Key type .llcrypt unregistered [ 1655.971617] Key type ._llcrypt unregistered [ 1683.262884] Key type ._llcrypt registered [ 1683.264761] Key type .llcrypt registered [ 1683.444387] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1684.523753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1684.693802] alg: No test for adler32 (adler32-zlib) [ 1685.723387] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1686.183347] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1687.944077] Key type lgssc registered [ 1689.032656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1698.063595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1698.090022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.578563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1705.212350] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug +malloc all [ 1710.062371] LustreError: 75606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1710.100754] LustreError: 75606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1715.181102] LustreError: 75607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.990724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.454227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1721.538124] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug +malloc all [ 1730.371526] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1737.860539] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1745.520793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1745.953179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.563513] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug +malloc all [ 1760.948974] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1769.576794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1780.193205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.193811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.202582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.209035] Lustre: Skipped 1 previous similar message [ 1785.317175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1785.324719] Lustre: Skipped 2 previous similar messages [ 1785.828661] Lustre: server umount lustre-OST0000 complete [ 1793.507729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1793.527556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1793.554242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.552974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.555331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.603028] Lustre: Skipped 1 previous similar message [ 1797.564622] Lustre: server umount lustre-MDT0000 complete [ 1800.672248] LustreError: 75606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.686952] LustreError: 75606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1802.251926] LustreError: 75585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265367 with bad export cookie 4730788412120529646 [ 1802.263514] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.281962] LustreError: 75585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1802.838212] Lustre: server umount lustre-MDT0001 complete [ 1817.118931] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1820.437922] Key type lgssc unregistered [ 1820.779734] LNet: 79160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.788543] LNetError: 79160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.806169] LNet: Removed LNI 192.168.202.151@tcp [ 1821.611409] Key type .llcrypt unregistered [ 1821.613500] Key type ._llcrypt unregistered [ 1847.774637] Key type ._llcrypt registered [ 1847.775893] Key type .llcrypt registered [ 1847.867592] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1849.327767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1849.351903] alg: No test for adler32 (adler32-zlib) [ 1850.513883] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1850.845616] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1852.632258] Key type lgssc registered [ 1853.908789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1862.124903] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1862.143658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.727786] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1882.849435] LustreError: 80544:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.151@tcp: failed processing log, type 4: rc = -110 [ 1909.535255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.282897] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1924.336667] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1931.579262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.648137] Lustre: *** cfs_fail_loc=706, val=0*** [ 1932.031568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.120502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1937.078245] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1937.381806] LustreError: 80570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.406687] LustreError: 80570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1942.504919] LustreError: 80569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.528235] LustreError: 82231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.558621] LustreError: 82231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1946.519039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.062322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1952.278635] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1961.139107] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1967.939757] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1972.711578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.716339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.743956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1976.754608] Lustre: server umount lustre-OST0000 complete [ 1982.944478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.956572] Lustre: Skipped 2 previous similar messages [ 1982.958985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.960773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1982.988279] Lustre: Skipped 2 previous similar messages [ 1987.037814] Lustre: server umount lustre-MDT0000 complete [ 1988.067557] LustreError: 82231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.098586] LustreError: 82231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1990.914617] LustreError: 81657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265556 with bad export cookie 8817560679159630142 [ 1990.923471] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.926442] LustreError: 81657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1991.202822] Lustre: server umount lustre-MDT0001 complete [ 2003.104090] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2006.120661] Key type lgssc unregistered [ 2006.477289] LNet: 84161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.490616] LNetError: 84161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.511349] LNet: Removed LNI 192.168.202.151@tcp [ 2007.427682] Key type .llcrypt unregistered [ 2007.429462] Key type ._llcrypt unregistered [ 2036.536409] Key type ._llcrypt registered [ 2036.539837] Key type .llcrypt registered [ 2036.665967] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2037.792457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2037.890980] alg: No test for adler32 (adler32-zlib) [ 2039.003398] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 2039.244421] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2040.875306] Key type lgssc registered [ 2042.524223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2051.665448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2051.716781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.918248] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2070.401985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.015458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2076.129163] LustreError: 86264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.154625] LustreError: 86264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2076.868840] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2081.252565] LustreError: 86267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.273270] LustreError: 86267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2086.374470] LustreError: 86268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.390941] LustreError: 86268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2086.761265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.346751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.515598] LustreError: 86267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.536192] LustreError: 86267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2094.123422] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2100.711149] LustreError: 86268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.732903] LustreError: 86268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2103.342853] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2103.359247] Lustre: MGS: Client e0b7da09-71c2-4dc6-a704-4e6bd696f7d5 (at 0@lo) reconnecting [ 2103.394361] LustreError: 86278:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2103.777591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.894779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2108.869421] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2125.283355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2125.291984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.326928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2127.192875] Lustre: server umount lustre-OST0000 complete [ 2134.501223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.511269] Lustre: Skipped 1 previous similar message [ 2134.511458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.533579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2134.535806] Lustre: Skipped 1 previous similar message [ 2139.617800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.627618] Lustre: Skipped 1 previous similar message [ 2141.688886] Lustre: server umount lustre-MDT0000 complete [ 2143.725495] LustreError: 87808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.743519] LustreError: 87808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2146.182274] Lustre: server umount lustre-MDT0001 complete [ 2150.614717] Lustre: server umount MGS complete [ 2163.821448] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:15:27 (1789265727) [ 2195.547435] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 2208.445580] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2255.874226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.516799] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2272.012258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.192824] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2272.257971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2272.373167] Lustre: lustre-MDT0001: new disk, initializing [ 2272.460092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2272.499291] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2272.508497] Lustre: 93451:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2272.518569] LustreError: 93451:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [ 2277.184585] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2287.936520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2288.322656] Lustre: lustre-OST0000: new disk, initializing [ 2288.338291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2288.360908] Lustre: 94218:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2288.418226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.773728] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2296.865018] Lustre: 94636:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000089c4d8e4 x1876180970022400/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789265878 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2304.560879] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2311.964513] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2311.972040] Lustre: MGS: Client 2f778547-59b7-4d78-9dc7-7bb108e9db0c (at 0@lo) reconnecting [ 2311.998139] Lustre: 95266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2312.007643] LustreError: 93436:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2312.016612] LustreError: 93436:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2312.041133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2312.149666] Lustre: lustre-MDT0000: new disk, initializing [ 2312.302547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.334647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2312.367424] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2313.450083] Lustre: 95290:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000004ea9c44 x1876180970029440/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789265894 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2317.551861] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2318.562888] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2322.545735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2322.558935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2322.650542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2348.521195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2348.529066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.559192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2353.815281] Lustre: server umount lustre-OST0000 complete [ 2361.829365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.834135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2361.845480] Lustre: Skipped 1 previous similar message [ 2361.846863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.846869] Lustre: Skipped 3 previous similar messages [ 2361.864150] LustreError: Skipped 1 previous similar message [ 2363.492910] Lustre: server umount lustre-MDT0000 complete [ 2366.948464] LustreError: 93420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.961907] LustreError: 93420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2373.833945] Lustre: server umount lustre-MDT0001 complete [ 2377.622790] Lustre: server umount MGS complete [ 2389.978293] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:19:13 (1789265953) [ 2410.910693] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2414.379334] Key type lgssc unregistered [ 2414.754870] LNet: 97792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.765833] LNetError: 97792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.790189] LNet: Removed LNI 192.168.202.151@tcp [ 2415.853931] Key type .llcrypt unregistered [ 2415.857262] Key type ._llcrypt unregistered [ 2437.581812] Key type ._llcrypt registered [ 2437.585550] Key type .llcrypt registered [ 2437.714185] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2438.837028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2438.876126] alg: No test for adler32 (adler32-zlib) [ 2440.146532] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 2440.402966] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2442.087184] Key type lgssc registered [ 2443.059948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2451.441557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2451.459094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.930198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.747652] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2463.208324] LustreError: 99020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.225295] LustreError: 99020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2468.322182] LustreError: 99021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.544276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.830721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2473.382440] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2481.014089] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2487.159837] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2494.474499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2494.794347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.852446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2501.215935] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2508.529738] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2523.617034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2523.623000] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.642905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2524.646513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2524.649873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2524.664992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.762686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.769869] Lustre: Skipped 1 previous similar message [ 2534.886731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.896209] Lustre: Skipped 2 previous similar messages [ 2535.394376] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2535.571183] Lustre: server umount lustre-OST0000 complete [ 2540.002968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.010189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.027508] Lustre: Skipped 1 previous similar message [ 2540.044618] Lustre: Skipped 1 previous similar message [ 2549.219399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2549.227487] Lustre: Skipped 5 previous similar messages [ 2554.335343] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2554.486602] Lustre: server umount lustre-MDT0000 complete [ 2558.219223] LustreError: 99001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266123 with bad export cookie 1463891932447561453 [ 2558.228420] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.237827] LustreError: 99001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2558.633905] Lustre: server umount lustre-MDT0001 complete [ 2568.785409] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2571.548708] Key type lgssc unregistered [ 2571.834852] LNet: 102576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.839372] LNetError: 102576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2571.851422] LNet: Removed LNI 192.168.202.151@tcp [ 2572.591899] Key type .llcrypt unregistered [ 2572.594342] Key type ._llcrypt unregistered [ 2589.744669] Key type ._llcrypt registered [ 2589.748320] Key type .llcrypt registered [ 2589.954834] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2591.148494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2591.226126] alg: No test for adler32 (adler32-zlib) [ 2592.383893] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 2592.656157] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2594.359219] Key type lgssc registered [ 2595.871336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2601.483478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2601.489939] LustreError: 103642:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2601.534342] LustreError: 103654:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2601.538940] LustreError: 103642:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4956 [ 2601.543272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.545439] LustreError: 103642:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2601.562715] LustreError: 103642:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2603.048710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.774181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.088727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2617.064874] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2622.662652] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing lsmod [ 2625.749806] LustreError: 104857:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2625.770515] LustreError: 104861:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2625.773855] LustreError: 104857:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 2625.777059] LustreError: 104857:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2625.781020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2625.784754] LustreError: 104857:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2626.160141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.219645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2632.673930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2632.688701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.700491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2633.704923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2633.707536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.717471] Lustre: Skipped 1 previous similar message [ 2633.740713] Lustre: Skipped 1 previous similar message [ 2635.264237] Lustre: server umount lustre-MDT0000 complete [ 2636.836946] LustreError: 103688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.888012] LustreError: 103688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2639.755146] LustreError: 103666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266205 with bad export cookie 7531491485519393852 [ 2639.777267] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.792884] Lustre: 103684:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2640.991918] LustreError: 104122:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000aa4bf967 x1876181550912384/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2641.200556] Lustre: server umount lustre-MDT0001 complete [ 2655.790808] Lustre: server umount lustre-OST0000 complete [ 2664.987197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.409380] LustreError: 105926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.518958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.195898] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2670.565640] LustreError: 105927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.681720] LustreError: 105926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.249655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.727183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.021574] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2693.848036] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2700.475875] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2707.668569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.011754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.094229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 2714.663193] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2722.628706] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2735.585705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2735.594950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.605812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2735.610734] Lustre: Skipped 1 previous similar message [ 2735.622058] Lustre: Skipped 1 previous similar message [ 2740.722907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.737253] Lustre: Skipped 2 previous similar messages [ 2741.719080] Lustre: server umount lustre-OST0000 complete [ 2746.338766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2746.347375] LustreError: Skipped 1 previous similar message [ 2746.350817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.365461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.950400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.074054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2756.095654] Lustre: Skipped 2 previous similar messages [ 2760.159665] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2760.503759] Lustre: server umount lustre-MDT0000 complete [ 2761.190393] LustreError: 105922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.207295] LustreError: 105922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2764.830553] LustreError: 105907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266330 with bad export cookie 7531491485519394776 [ 2764.839388] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.850440] LustreError: 105907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2765.333498] Lustre: server umount lustre-MDT0001 complete [ 2775.316780] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2777.685587] Key type lgssc unregistered [ 2777.982743] LNet: 109480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2777.990667] LNetError: 109480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2779.046891] LNet: Removed LNI 192.168.202.151@tcp [ 2779.922793] Key type .llcrypt unregistered [ 2779.925923] Key type ._llcrypt unregistered [ 2803.419903] Key type ._llcrypt registered [ 2803.422201] Key type .llcrypt registered [ 2803.508616] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2804.416260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2804.520180] alg: No test for adler32 (adler32-zlib) [ 2805.560982] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 2805.748662] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2807.397784] Key type lgssc registered [ 2808.406321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2814.722488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2814.733376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.040683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.386932] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2826.211532] LustreError: 110897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.220388] LustreError: 110897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2828.933672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.311165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2834.035499] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2842.412447] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2848.651893] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2855.417369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2855.834522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.973870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2862.118665] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2869.364938] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2878.586926] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2881.970353] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2885.603218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2885.607194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.624804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.633941] Lustre: Skipped 1 previous similar message [ 2885.647907] Lustre: Skipped 1 previous similar message [ 2890.725629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2890.728963] Lustre: Skipped 1 previous similar message [ 2891.266478] Lustre: server umount lustre-OST0000 complete [ 2895.844294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2895.845590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.853051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.857470] Lustre: Skipped 1 previous similar message [ 2895.864987] Lustre: Skipped 1 previous similar message [ 2900.837866] Lustre: server umount lustre-MDT0000 complete [ 2900.965167] LustreError: 110898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.434148] LustreError: 110877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266470 with bad export cookie 4112598669932645958 [ 2905.438078] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2905.442929] LustreError: 110877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2905.873456] Lustre: server umount lustre-MDT0001 complete [ 2920.200885] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2923.526216] Key type lgssc unregistered [ 2923.882609] LNet: 114536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2923.886886] LNetError: 114536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2923.899855] LNet: Removed LNI 192.168.202.151@tcp [ 2924.645347] Key type .llcrypt unregistered [ 2924.647878] Key type ._llcrypt unregistered [ 2942.834458] Key type ._llcrypt registered [ 2942.839347] Key type .llcrypt registered [ 2943.025170] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2943.980891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2944.021835] alg: No test for adler32 (adler32-zlib) [ 2945.123877] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 2945.417295] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2947.135657] Key type lgssc registered [ 2948.213743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2957.516773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2957.569961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.032446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.104889] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2968.555049] LustreError: 115760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.589988] LustreError: 115760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2973.680995] LustreError: 115760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.825753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.144624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2979.187151] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2987.784146] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2994.053615] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3001.485718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3001.766432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.835258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3009.030745] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3018.782510] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3029.479278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3029.486474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3029.487124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3029.505429] Lustre: Skipped 1 previous similar message [ 3029.533870] Lustre: Skipped 1 previous similar message [ 3034.600556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3034.612588] Lustre: Skipped 1 previous similar message [ 3035.074146] Lustre: server umount lustre-OST0000 complete [ 3039.713602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.730526] Lustre: Skipped 1 previous similar message [ 3039.732906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3039.738118] Lustre: Skipped 1 previous similar message [ 3044.832174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3044.837779] Lustre: Skipped 1 previous similar message [ 3045.677534] Lustre: server umount lustre-MDT0000 complete [ 3049.609581] LustreError: 115741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266615 with bad export cookie 13559385372403255194 [ 3049.614707] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3049.617692] LustreError: 115741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3049.996707] Lustre: server umount lustre-MDT0001 complete [ 3061.642247] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3064.295228] Key type lgssc unregistered [ 3064.539677] LNet: 119291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3064.546457] LNetError: 119291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3064.564187] LNet: Removed LNI 192.168.202.151@tcp [ 3065.371579] Key type .llcrypt unregistered [ 3065.373941] Key type ._llcrypt unregistered [ 3088.724660] Key type ._llcrypt registered [ 3088.727667] Key type .llcrypt registered [ 3088.862808] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3090.181690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3090.319058] alg: No test for adler32 (adler32-zlib) [ 3091.399223] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 3091.682615] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3093.401344] Key type lgssc registered [ 3094.754280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3102.967171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3102.984594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.454984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3109.217963] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3114.038035] LustreError: 120693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.059953] LustreError: 120693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3118.146819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.510982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3122.527245] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3130.075294] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3136.384386] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3144.462166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3144.913559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3149.002681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3152.704322] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3162.565672] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3171.124808] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3174.864120] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3174.868296] Lustre: Skipped 2 previous similar messages [ 3178.087351] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3178.127533] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3180.031886] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3180.040902] Lustre: Skipped 2 previous similar messages [ 3184.694603] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3184.701745] Lustre: Skipped 1 previous similar message [ 3189.645628] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.51@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3189.656626] Lustre: Skipped 2 previous similar messages [ 3192.318339] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.51@tcp 192.168.0.[2,10]@tcp [ 3202.436867] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.51/24@tcp in log lustre-MDT0000 [ 3202.457264] Lustre: Skipped 2 previous similar messages [ 3210.361233] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.51/24@tcp [ 3210.364738] Lustre: Skipped 1 previous similar message [ 3219.936276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3219.943151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.958288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3219.995937] Lustre: Skipped 1 previous similar message [ 3223.451371] Lustre: server umount lustre-OST0000 complete [ 3230.177779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.213100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.226681] Lustre: Skipped 2 previous similar messages [ 3234.603428] Lustre: server umount lustre-MDT0000 complete [ 3235.301174] LustreError: 120688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.335476] LustreError: 120688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3239.259555] LustreError: 120672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266804 with bad export cookie 5301054996960375392 [ 3239.260246] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3239.264381] LustreError: 120672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3239.518386] Lustre: server umount lustre-MDT0001 complete [ 3254.225979] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3258.023485] Key type lgssc unregistered [ 3258.478296] LNet: 125589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3258.491576] LNetError: 125589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3258.519427] LNet: Removed LNI 192.168.202.151@tcp [ 3259.788274] Key type .llcrypt unregistered [ 3259.792931] Key type ._llcrypt unregistered [ 3296.231843] Key type ._llcrypt registered [ 3296.233861] Key type .llcrypt registered [ 3296.337170] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3297.836631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3297.908094] alg: No test for adler32 (adler32-zlib) [ 3299.022157] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 3299.277542] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3300.953123] Key type lgssc registered [ 3302.203852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3310.567472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3310.586269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.195856] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3312.215963] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.51/24@tcp [ 3312.237443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.979705] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3322.342575] LustreError: 127150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3322.361255] LustreError: 127150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3326.309329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.775268] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3326.829955] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.51/24@tcp [ 3326.871550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3332.773386] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3341.636210] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3348.253208] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3357.041573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3357.448697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.986773] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3371.907481] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3382.026757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3395.066608] Lustre: server umount lustre-OST0000 complete [ 3402.735195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.743433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.743436] Lustre: Skipped 1 previous similar message [ 3402.753442] Lustre: Skipped 1 previous similar message [ 3407.845986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3407.856446] Lustre: Skipped 1 previous similar message [ 3408.967831] Lustre: server umount lustre-MDT0000 complete [ 3412.964708] LustreError: 127149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3412.969906] LustreError: 127149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3413.168180] LustreError: 127130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266978 with bad export cookie 7909520375146986807 [ 3413.192131] LustreError: 127130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3413.201112] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3422.853940] Lustre: server umount lustre-MDT0001 complete [ 3433.382173] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3435.678812] Key type lgssc unregistered [ 3435.965527] LNet: 130763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3435.974985] LNetError: 130763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3435.993420] LNet: Removed LNI 192.168.202.151@tcp [ 3436.768794] Key type .llcrypt unregistered [ 3436.773113] Key type ._llcrypt unregistered [ 3463.418166] Key type ._llcrypt registered [ 3463.420161] Key type .llcrypt registered [ 3463.501599] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 3480.965708] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3482.015475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3482.134257] alg: No test for adler32 (adler32-zlib) [ 3483.265255] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 3483.492873] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3485.167426] Key type lgssc registered [ 3486.680131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3524.694242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3524.725457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.005271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3526.045336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3526.122232] Lustre: lustre-MDT0000: new disk, initializing [ 3526.246892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.282598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3530.507107] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3541.299748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3541.448311] Lustre: 134157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3541.506214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3541.509708] Lustre: Skipped 1 previous similar message [ 3541.519554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3541.598261] Lustre: lustre-MDT0001: new disk, initializing [ 3541.672993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3541.705377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3541.717918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3546.187481] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3555.265758] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3561.895674] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3571.457202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3572.025609] Lustre: lustre-OST0000: new disk, initializing [ 3572.042995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3572.069688] Lustre: 136128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3572.296659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.555700] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3581.563130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3581.600752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3581.789762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3594.819484] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3600.817704] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3600.995934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3605.934178] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3606.217410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3612.647375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3612.655551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.663443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3612.671226] Lustre: Skipped 1 previous similar message [ 3612.680477] Lustre: Skipped 1 previous similar message [ 3614.423570] Lustre: server umount lustre-OST0000 complete [ 3618.784558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3618.793696] LustreError: Skipped 1 previous similar message [ 3618.796971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.806826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3622.881127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3622.883670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3622.899843] Lustre: Skipped 1 previous similar message [ 3624.674795] Lustre: server umount lustre-MDT0000 complete [ 3628.430063] LustreError: 134150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267193 with bad export cookie 3076713870799461547 [ 3628.436117] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3628.443662] LustreError: 134150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3628.904762] Lustre: server umount lustre-MDT0001 complete [ 3637.430090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3637.823368] LustreError: 138344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.844101] LustreError: 138344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3637.896149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.343406] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3643.367741] LustreError: 138345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3648.487177] LustreError: 138344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.761311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.058401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3657.370529] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3665.351783] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3671.036563] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3678.475282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3678.811220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3685.641153] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3694.078571] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3699.413815] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3699.682805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3705.316951] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3705.527109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3716.395903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3716.642638] Lustre: lustre-OST0001: new disk, initializing [ 3716.646864] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3716.657791] Lustre: 141717:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3716.767086] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3724.461945] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3725.481301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3725.501753] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3725.676503] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3733.989942] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3734.222824] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3740.237338] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3740.630390] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3746.871489] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fa0cc9cc000.ost_server_uuid 50 [ 3748.711664] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fa0cc9cc000.ost_server_uuid in IDLE state after 0 sec [ 3761.634384] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.635144] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3761.672316] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3761.689661] Lustre: Skipped 1 previous similar message [ 3765.945581] Lustre: server umount lustre-OST0001 complete [ 3766.773422] LustreError: 140168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3766.805435] LustreError: 140168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3770.339184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3770.339771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3770.347643] LustreError: Skipped 1 previous similar message [ 3770.351837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3770.368329] Lustre: Skipped 2 previous similar messages [ 3770.401109] Lustre: Skipped 1 previous similar message [ 3771.878023] LustreError: 142136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.891830] LustreError: 142136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3776.290511] Lustre: server umount lustre-OST0000 complete [ 3781.410689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.214865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3786.721800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3786.728287] Lustre: Skipped 4 previous similar messages [ 3788.048309] Lustre: server umount lustre-MDT0000 complete [ 3791.845033] LustreError: 142923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.874781] LustreError: 142923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3792.539517] LustreError: 138324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267358 with bad export cookie 3076713870799462429 [ 3792.544204] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.550160] LustreError: 138324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3793.087253] Lustre: server umount lustre-MDT0001 complete [ 3815.174820] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3817.964491] Key type lgssc unregistered [ 3818.347218] LNet: 144721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3818.376898] LNetError: 144721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3818.404159] LNet: Removed LNI 192.168.202.151@tcp [ 3819.447156] Key type .llcrypt unregistered [ 3819.453255] Key type ._llcrypt unregistered [ 3866.455463] Key type ._llcrypt registered [ 3866.457842] Key type .llcrypt registered [ 3866.571697] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 3882.287367] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3883.372186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3883.393763] alg: No test for adler32 (adler32-zlib) [ 3884.461348] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 3884.726652] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3886.383381] Key type lgssc registered [ 3887.785584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3925.438857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3925.463767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.816441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3926.859922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3926.974461] Lustre: lustre-MDT0000: new disk, initializing [ 3927.110620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.132784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3932.318859] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3943.211438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.428337] Lustre: 148875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3943.503487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3943.509117] Lustre: Skipped 1 previous similar message [ 3943.592668] Lustre: lustre-MDT0001: new disk, initializing [ 3943.733468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3943.790468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3943.803546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3948.718900] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3958.804629] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3964.672614] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3970.991073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3971.314758] Lustre: lustre-OST0000: new disk, initializing [ 3971.319183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3971.326862] Lustre: 150841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3971.440236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.190049] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3980.342749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3980.348756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3980.446975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3986.544972] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3991.444845] Lustre: Failing over lustre-OST0000 [ 3991.652594] Lustre: server umount lustre-OST0000 complete [ 4010.463305] Lustre: 147096:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267559/real 1789267559] req@00000000a0525549 x1876182905684224/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789267575 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4010.510220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.079096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.512749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.531633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.539756] Lustre: 147095:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267561/real 1789267561] req@000000005a3e6371 x1876182905685376/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789267577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4011.562440] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4012.588969] Lustre: 147098:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267561/real 1789267561] req@00000000e27e34a4 x1876182905685504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789267577 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4015.525967] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4016.608398] Lustre: 147097:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267566/real 1789267566] req@00000000c3ac7b49 x1876182905686528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789267582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4016.647191] Lustre: 147097:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4018.551784] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4021.038765] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4021.039025] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4021.728894] Lustre: 147097:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267571/real 1789267571] req@000000009ec45405 x1876182905687680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789267587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4021.751490] Lustre: 147097:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4023.460322] Lustre: Failing over lustre-MDT0000 [ 4025.832834] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4025.838492] Lustre: server umount lustre-MDT0000 complete [ 4025.856364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4030.949034] LustreError: 148885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.987626] LustreError: 148885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4036.066972] LustreError: 151520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.095333] LustreError: 151520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4041.192457] LustreError: 151017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4041.215567] LustreError: 151017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4041.679164] Lustre: 147098:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789267591/real 1789267591] req@00000000dd711442 x1876182905697536/t0(0) o400->MGC192.168.202.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789267607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4041.708568] Lustre: 147098:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4041.718911] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4046.666872] LustreError: 148880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.689204] LustreError: 148880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4048.070887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.756464] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.790423] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4052.230568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4052.295343] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4056.822635] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4057.596560] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4057.604733] Lustre: Skipped 1 previous similar message [ 4057.637534] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4057.700839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4058.242071] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4067.813317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.830321] Lustre: Skipped 2 previous similar messages [ 4067.851951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.417432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4072.425813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.447632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4073.898658] Lustre: server umount lustre-OST0000 complete [ 4083.169339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4083.173069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.188773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4083.202880] Lustre: Skipped 1 previous similar message [ 4083.209521] Lustre: Skipped 1 previous similar message [ 4085.871335] Lustre: server umount lustre-MDT0000 complete [ 4088.289404] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.313869] LustreError: 148882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4089.541858] LustreError: 148868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267655 with bad export cookie 9452758670866465080 [ 4089.542739] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4089.564815] LustreError: 148868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4089.935805] Lustre: server umount lustre-MDT0001 complete [ 4100.812580] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4103.989812] Key type lgssc unregistered [ 4104.275852] LNet: 154498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4104.286333] LNetError: 154498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4104.317227] LNet: Removed LNI 192.168.202.151@tcp [ 4105.527413] Key type .llcrypt unregistered [ 4105.531373] Key type ._llcrypt unregistered [ 4145.228903] Key type ._llcrypt registered [ 4145.231630] Key type .llcrypt registered [ 4145.328626] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4146.602237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4146.644862] alg: No test for adler32 (adler32-zlib) [ 4147.698873] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 4147.930530] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4149.615339] Key type lgssc registered [ 4150.825791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4159.159607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4159.184473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.743198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.627755] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4171.235704] LustreError: 156015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.276079] LustreError: 156015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4176.356665] LustreError: 156014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.652290] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4181.475239] LustreError: 156015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.604699] LustreError: 156044:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4182.617776] LustreError: 156044:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4182.623950] LustreError: 156044:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4183.136391] Lustre: server umount lustre-MDT0000 complete [ 4201.216089] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4212.104949] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4254.464044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.816662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4254.904255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4255.120589] Lustre: lustre-MDT0000: new disk, initializing [ 4255.242927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.287453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4263.044728] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4274.963820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.088680] Lustre: 159348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4275.146360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4275.150330] Lustre: Skipped 1 previous similar message [ 4275.226548] Lustre: lustre-MDT0001: new disk, initializing [ 4275.352432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4275.415891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4275.431375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4280.580492] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4290.784157] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4296.956074] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4303.545056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4303.805564] Lustre: lustre-OST0000: new disk, initializing [ 4303.809835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4303.818870] Lustre: 161321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4303.892793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.513440] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4309.607514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4309.622696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4309.800460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4319.501837] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4323.405668] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4324.983898] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:51:28 (1789267888) [ 4582.894085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.908041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.922062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.465073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.473027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.485529] Lustre: Skipped 1 previous similar message [ 4586.499609] Lustre: Skipped 2 previous similar messages [ 4588.826854] Lustre: server umount lustre-MDT0000 complete [ 4591.585049] LustreError: 159361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.599853] LustreError: 159361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4592.177771] LustreError: 159342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268157 with bad export cookie 10098019537986471475 [ 4592.181988] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4592.187803] LustreError: 159342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4596.704876] LustreError: 159357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.705347] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.750262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4598.785846] Lustre: server umount lustre-MDT0001 complete [ 4617.183186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4617.456313] Lustre: server umount lustre-OST0000 complete [ 4628.354857] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4639.154680] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4678.352132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4678.732392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4678.808655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4678.902406] Lustre: lustre-MDT0000: new disk, initializing [ 4678.950371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.981860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4683.924540] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4697.682571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.873788] Lustre: 165985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4697.938050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4697.944306] Lustre: Skipped 1 previous similar message [ 4698.055356] Lustre: lustre-MDT0001: new disk, initializing [ 4698.246657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4698.331229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4698.356217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4703.991348] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4714.688165] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4721.848884] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4729.643711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4730.015321] Lustre: lustre-OST0000: new disk, initializing [ 4730.020514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4730.030595] Lustre: 167954:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4730.129793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4737.596861] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4738.695160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4738.712770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4738.920255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4748.752533] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4753.855331] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4754.164333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4759.065714] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4759.345522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4764.645686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4764.651702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4764.663057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4764.663065] Lustre: Skipped 1 previous similar message [ 4764.678060] Lustre: Skipped 1 previous similar message [ 4764.722406] LustreError: Skipped 1 previous similar message [ 4768.093341] Lustre: server umount lustre-OST0000 complete [ 4774.887756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4774.899914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4774.917288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4774.929505] Lustre: Skipped 1 previous similar message [ 4774.941192] Lustre: Skipped 1 previous similar message [ 4778.418458] Lustre: server umount lustre-MDT0000 complete [ 4780.000097] LustreError: 169496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.013679] LustreError: 169496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4782.550841] LustreError: 165977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268348 with bad export cookie 10098019537986630305 [ 4782.562256] LustreError: 165977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4782.565227] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.979807] Lustre: server umount lustre-MDT0001 complete [ 4792.198483] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:59:15 (1789268355) [ 4809.745265] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4819.761448] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4863.498152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.845257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4863.902785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4864.065783] Lustre: lustre-MDT0000: new disk, initializing [ 4864.179404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4864.205761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4869.862792] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4882.544166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4882.709911] Lustre: 172714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4883.110342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4889.135909] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4900.019483] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4906.335473] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4913.322020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.676469] Lustre: lustre-OST0000: new disk, initializing [ 4913.679882] Lustre: Skipped 1 previous similar message [ 4913.684471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4913.693992] Lustre: Skipped 2 previous similar messages [ 4913.699690] Lustre: 174704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4919.862333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4919.873342] Lustre: Skipped 1 previous similar message [ 4919.884090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4919.988794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4921.294417] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4932.258767] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4940.778047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.781748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4940.788735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4940.794254] Lustre: Skipped 1 previous similar message [ 4940.834341] LustreError: Skipped 1 previous similar message [ 4954.079313] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4954.452807] Lustre: server umount lustre-OST0000 complete [ 4959.715867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4959.730223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.739239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4959.753137] Lustre: Skipped 5 previous similar messages [ 4965.218936] Lustre: server umount lustre-MDT0000 complete [ 4966.373229] LustreError: 172720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4966.383985] LustreError: 172720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4968.795067] LustreError: 172705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268534 with bad export cookie 10098019537986631187 [ 4968.801578] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4968.808308] LustreError: 172705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4969.262705] Lustre: server umount lustre-MDT0001 complete [ 4980.999440] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:02:24 (1789268544) [ 5000.814438] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 5012.882518] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5059.642580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.993532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5060.037215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5060.123888] Lustre: lustre-MDT0000: new disk, initializing [ 5060.261294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.265937] Lustre: Skipped 2 previous similar messages [ 5060.289343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5064.636216] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5074.892785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.015966] Lustre: 179032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5075.269712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5079.467621] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5089.886328] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5096.396902] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5103.994434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5104.352416] Lustre: 181006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5110.778944] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5111.856621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5112.013475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5121.184310] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5132.772629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5132.774797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.784917] LustreError: Skipped 1 previous similar message [ 5132.809272] Lustre: Skipped 1 previous similar message [ 5132.811566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.811576] Lustre: Skipped 2 previous similar messages [ 5135.256442] Lustre: server umount lustre-OST0000 complete [ 5145.713437] Lustre: server umount lustre-MDT0000 complete [ 5148.129274] LustreError: 179043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5148.173080] LustreError: 179043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5150.379628] LustreError: 179023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268715 with bad export cookie 10098019537986632272 [ 5150.383081] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5150.403654] LustreError: 179023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5151.055769] Lustre: server umount lustre-MDT0001 complete [ 5164.263375] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 5168.100959] Key type lgssc unregistered [ 5168.416547] LNet: 182860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5168.420165] LNetError: 182860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5168.453723] LNet: Removed LNI 192.168.202.151@tcp [ 5169.568844] Key type .llcrypt unregistered [ 5169.574935] Key type ._llcrypt unregistered [ 5203.951234] Key type ._llcrypt registered [ 5203.956489] Key type .llcrypt registered [ 5204.206545] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5205.617589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5205.815286] alg: No test for adler32 (adler32-zlib) [ 5207.806583] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 5208.270702] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 5210.055521] Key type lgssc registered [ 5212.110752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5223.120356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5223.153260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.740848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5230.827462] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5235.171047] LustreError: 184274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.200782] LustreError: 184274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5240.294404] LustreError: 184273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.421831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.672601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5245.805397] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5255.712585] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5263.514560] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5271.918359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5272.166305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.224561] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5289.904693] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5294.712972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5315.556594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5315.582961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5319.996153] Lustre: server umount lustre-OST0000 complete [ 5325.793404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.814117] Lustre: Skipped 1 previous similar message [ 5325.833307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5325.841415] Lustre: Skipped 2 previous similar messages [ 5330.608905] Lustre: server umount lustre-MDT0000 complete [ 5330.914075] LustreError: 184269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.948493] LustreError: 184269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5334.624428] LustreError: 184254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789268900 with bad export cookie 14529206461124861378 [ 5334.637197] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5334.647963] LustreError: 184254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5335.155955] Lustre: server umount lustre-MDT0001 complete [ 5346.136669] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 5349.824951] Key type lgssc unregistered [ 5350.190817] LNet: 187836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5350.195732] LNetError: 187836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5350.213950] LNet: Removed LNI 192.168.202.151@tcp [ 5351.238114] Key type .llcrypt unregistered [ 5351.244675] Key type ._llcrypt unregistered [ 5378.711973] Key type ._llcrypt registered [ 5378.716737] Key type .llcrypt registered [ 5378.880685] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5380.269985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5380.357430] alg: No test for adler32 (adler32-zlib) [ 5381.398370] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 5381.655624] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 5383.335207] Key type lgssc registered [ 5384.629393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5393.298217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5393.335266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.982471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5400.171518] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5405.160506] LustreError: 189249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5405.187568] LustreError: 189249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5410.066604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.275154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5410.464557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5415.102610] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5425.088360] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5432.367305] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5439.836688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5440.283938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5441.344904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5447.448298] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5457.136895] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5465.057755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5465.078624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5465.091340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5465.102053] Lustre: Skipped 1 previous similar message [ 5465.120331] Lustre: Skipped 1 previous similar message [ 5470.183590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5470.193280] Lustre: Skipped 2 previous similar messages [ 5471.214985] Lustre: server umount lustre-OST0000 complete [ 5479.555773] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9fa0cde4d800.ost_server_uuid 50 [ 5481.376542] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5497.313997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5497.327396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.347708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5497.357902] Lustre: Skipped 1 previous similar message [ 5499.882611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.449614] Lustre: server umount lustre-MDT0000 complete [ 5504.993442] LustreError: 191689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.031120] LustreError: 191689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5507.522387] LustreError: 189229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269073 with bad export cookie 3298752493682132797 [ 5507.525872] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5507.548645] LustreError: 189229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5507.962688] Lustre: server umount lustre-MDT0001 complete [ 5517.898315] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:11:21 (1789269081) [ 5524.690559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.065881] LustreError: 193171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.138732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5530.108679] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5530.595385] LustreError: 193172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5534.691580] LustreError: 193171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.015293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.633494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5545.118456] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5553.523221] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5561.176936] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5569.086807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5569.519439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.582261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5576.632349] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5585.908879] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5594.172667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5594.321320] Lustre: lustre-OST0001: new disk, initializing [ 5594.324550] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5594.329518] Lustre: 196013:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5594.394703] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5600.747295] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5601.953600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5602.000046] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5607.423546] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5610.330643] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5616.420547] Lustre: lustre-OST0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 5616.427567] Lustre: Skipped 2 previous similar messages [ 5616.610137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5616.614535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.620301] LustreError: Skipped 1 previous similar message [ 5616.630744] Lustre: Skipped 1 previous similar message [ 5619.755722] Lustre: server umount lustre-OST0000 complete [ 5621.489024] LustreError: 195861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.508399] LustreError: 195861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5626.018157] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5626.198397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5631.908246] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5632.320310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5641.199620] LustreError: 194993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.220546] LustreError: 194993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5646.314791] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.338214] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5646.340616] Lustre: Skipped 4 previous similar messages [ 5651.405593] Lustre: server umount lustre-OST0001 complete [ 5656.546236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.558374] Lustre: Skipped 2 previous similar messages [ 5661.789344] Lustre: server umount lustre-MDT0000 complete [ 5665.379740] LustreError: 197478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269230 with bad export cookie 3298752493682133861 [ 5665.387254] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5665.393099] LustreError: 197478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5665.653353] Lustre: server umount lustre-MDT0001 complete [ 5690.711757] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:14:14 (1789269254) [ 5697.419741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.833394] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5697.885402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5698.213843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.273797] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5712.532063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5712.681820] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5712.769840] Lustre: 199451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5717.910790] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5726.139361] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5731.232738] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5738.092824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5738.249736] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5744.945476] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5748.904341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5753.804068] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5762.122603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5762.202905] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5762.464194] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5762.468972] Lustre: Skipped 2 previous similar messages [ 5770.142570] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5778.612670] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5784.843484] Lustre: lustre-OST0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 5784.850715] Lustre: Skipped 2 previous similar messages [ 5785.067955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5785.076981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.080183] LustreError: Skipped 1 previous similar message [ 5785.101231] Lustre: Skipped 1 previous similar message [ 5796.320883] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5796.491641] Lustre: server umount lustre-OST0000 complete [ 5800.173523] LustreError: 201304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.211131] LustreError: 201304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5816.292949] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5816.304199] LustreError: Skipped 1 previous similar message [ 5818.676344] Lustre: server umount lustre-OST0001 complete [ 5826.017555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5826.027181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.044152] Lustre: Skipped 2 previous similar messages [ 5826.059282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5826.073193] Lustre: Skipped 10 previous similar messages [ 5828.561030] Lustre: server umount lustre-MDT0000 complete [ 5832.175567] LustreError: 200197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269397 with bad export cookie 3298752493682135275 [ 5832.180737] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5832.191342] LustreError: 200197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5832.527204] Lustre: server umount lustre-MDT0001 complete [ 5853.988414] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:16:57 (1789269417) [ 5868.993588] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 5877.606545] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5915.435665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5915.786920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5915.817433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5915.907551] Lustre: lustre-MDT0000: new disk, initializing [ 5916.000127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5916.042705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5920.729866] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5930.911562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.030711] Lustre: 207549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5931.067959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5931.071931] Lustre: Skipped 1 previous similar message [ 5931.136685] Lustre: lustre-MDT0001: new disk, initializing [ 5931.320829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5931.334255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5935.226845] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5943.679969] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5949.149964] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5955.652880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5955.853412] Lustre: lustre-OST0000: new disk, initializing [ 5955.859577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5955.865501] Lustre: 209517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5961.945858] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5963.816550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5963.833927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5963.934749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5970.668293] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5975.232615] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5975.457801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5979.346439] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5979.518338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5984.738576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5984.740418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.746575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5984.746584] Lustre: Skipped 1 previous similar message [ 5984.765160] Lustre: Skipped 2 previous similar messages [ 5987.390944] Lustre: server umount lustre-OST0000 complete [ 5997.638238] Lustre: server umount lustre-MDT0000 complete [ 6000.100915] LustreError: 207560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.117245] LustreError: 207560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6000.904762] LustreError: 207540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269566 with bad export cookie 3298752493682136843 [ 6000.909750] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6000.911110] LustreError: 207540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6001.274513] Lustre: server umount lustre-MDT0001 complete [ 6009.170778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.911490] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6022.222854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6026.558076] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6034.527666] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6041.128179] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6048.758523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6056.193698] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6063.725502] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6067.777651] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6068.057181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6072.862382] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6073.034991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6075.889864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6075.899804] LustreError: Skipped 2 previous similar messages [ 6081.438394] Lustre: server umount lustre-OST0000 complete [ 6087.899861] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6093.065313] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6122.649465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6127.267151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.303515] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6130.530883] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6140.644878] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6141.746102] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.764028] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6142.259412] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6147.045816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.054369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.061837] LustreError: Skipped 1 previous similar message [ 6147.063207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.063213] Lustre: Skipped 7 previous similar messages [ 6147.095572] Lustre: Skipped 5 previous similar messages [ 6152.576555] Lustre: server umount lustre-OST0000 complete [ 6162.847185] Lustre: server umount lustre-MDT0000 complete [ 6166.591299] LustreError: 211716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269732 with bad export cookie 3298752493682137718 [ 6166.593955] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6166.606457] LustreError: 211716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6167.076505] Lustre: server umount lustre-MDT0001 complete [ 6176.277126] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:22:19 (1789269739) [ 6184.699561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6185.234631] LustreError: 217701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.252924] LustreError: 217701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6185.322044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6185.328224] Lustre: Skipped 6 previous similar messages [ 6190.044698] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6199.395198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.858090] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6214.339617] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6220.872560] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6227.637612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6234.665384] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6242.458106] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6248.211710] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6248.544982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6254.823342] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6255.038058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6263.322779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6263.514523] Lustre: lustre-OST0001: new disk, initializing [ 6263.517424] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6263.522278] Lustre: 221075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6270.944639] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6271.088380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6271.099814] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6271.228817] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6281.769401] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6288.094620] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6288.407648] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6293.298144] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6293.544543] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6297.079673] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6310.367122] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6310.511664] Lustre: server umount lustre-OST0001 complete [ 6315.831966] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6316.037780] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6321.232852] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6321.437598] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6352.594487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6357.994518] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6357.999133] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6357.999582] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6360.635332] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6370.806615] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6373.169395] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6381.846431] Lustre: server umount lustre-OST0001 complete [ 6404.065736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.086816] Lustre: Skipped 6 previous similar messages [ 6404.103505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6404.115191] Lustre: Skipped 15 previous similar messages [ 6405.956258] Lustre: server umount lustre-OST0000 complete [ 6424.031122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6424.210294] Lustre: server umount lustre-MDT0000 complete [ 6428.018515] LustreError: 217681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789269993 with bad export cookie 3298752493682138684 [ 6428.028285] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6428.034094] LustreError: 217681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6434.572363] Lustre: server umount lustre-MDT0001 complete [ 6458.485762] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:27:02 (1789270022) [ 6464.331806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6464.554857] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6464.583347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6469.113720] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6477.273948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.420395] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6477.477135] Lustre: 226324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6482.550373] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6490.246556] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6496.055860] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6503.076935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6503.413605] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6510.368961] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6518.766930] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6528.857819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6528.923765] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6535.687590] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6544.190964] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6545.941277] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6550.280205] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6550.446779] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6554.882540] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6555.136208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6560.543582] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fa0c4530000.ost_server_uuid 50 [ 6562.261985] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fa0c4530000.ost_server_uuid in IDLE state after 0 sec [ 6563.243309] Lustre: Permanently deactivating lustre-OST0001 [ 6563.270161] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6566.699670] Lustre: Permanently reactivating lustre-OST0001 [ 6572.518567] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6572.525545] LustreError: Skipped 2 previous similar messages [ 6577.585879] Lustre: server umount lustre-OST0001 complete [ 6577.636109] LustreError: 228169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6577.648637] LustreError: 228169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 35 previous similar messages [ 6586.831375] Lustre: server umount lustre-OST0000 complete [ 6596.468756] Lustre: server umount lustre-MDT0000 complete [ 6599.517356] LustreError: 230733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270165 with bad export cookie 3298752493682139979 [ 6599.517356] LustreError: 226316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270165 with bad export cookie 3298752493682139979 [ 6599.518400] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6599.524370] LustreError: 230733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6599.901755] Lustre: server umount lustre-MDT0001 complete [ 6627.248299] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:29:50 (1789270190) [ 6634.918078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.148329] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6635.172961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6635.181093] Lustre: Skipped 5 previous similar messages [ 6640.352804] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6650.387712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6650.502652] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6650.583051] Lustre: 233019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6655.054463] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6662.428775] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6668.347760] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6674.972694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6675.233703] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6675.258579] Lustre: Permanently deactivating lustre-OST0000 [ 6675.283583] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6681.120895] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6681.240805] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6689.328532] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6696.297409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6696.354159] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6702.277385] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6702.628065] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6709.886491] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6711.483584] Lustre: Permanently reactivating lustre-OST0000 [ 6711.493788] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6711.498544] Lustre: Skipped 2 previous similar messages [ 6717.526875] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6717.528303] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 00000000b0418ea6, cur 1789270283 deadline 1789270276 last 1789270246 [ 6717.544253] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6717.587911] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6717.592914] Lustre: Skipped 1 previous similar message [ 6717.650267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6732.090044] Lustre: server umount lustre-OST0001 complete [ 6750.175943] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6750.372575] Lustre: server umount lustre-OST0000 complete [ 6760.464993] Lustre: server umount lustre-MDT0000 complete [ 6764.161734] LustreError: 233011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270329 with bad export cookie 3298752493682141659 [ 6764.164257] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6764.174844] LustreError: 233011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6764.440094] Lustre: server umount lustre-MDT0001 complete [ 6774.810360] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 6777.324968] Key type lgssc unregistered [ 6777.580487] LNet: 237897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6777.589880] LNetError: 237897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6777.625805] LNet: Removed LNI 192.168.202.151@tcp [ 6778.515304] Key type .llcrypt unregistered [ 6778.517898] Key type ._llcrypt unregistered [ 6801.019819] Key type ._llcrypt registered [ 6801.022273] Key type .llcrypt registered [ 6801.091616] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 6801.934875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6801.953887] alg: No test for adler32 (adler32-zlib) [ 6803.025859] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 6803.202842] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 6804.895620] Key type lgssc registered [ 6805.804753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6814.759940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6814.785914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.170901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6819.656240] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6826.471052] LustreError: 239340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6826.503667] LustreError: 239340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6826.928310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6827.062672] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6827.094753] Lustre: 239328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6827.116029] Lustre: Permanently deactivating lustre-MDT0001 [ 6827.123478] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6827.384765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6831.613218] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6835.725608] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6839.178930] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6844.794954] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6850.800741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6851.116177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6852.137382] LustreError: 241182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.173566] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6856.876553] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6857.187734] LustreError: 241183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.286917] LustreError: 241559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.317155] LustreError: 241559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6861.330472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6863.600171] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6866.407912] LustreError: 241182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6866.427790] LustreError: 241182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6869.463362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6869.623959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6875.111576] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6875.127948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6875.579815] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6883.619483] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6887.797976] Lustre: Permanently reactivating lustre-MDT0001 [ 6887.815357] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6887.827702] Lustre: Skipped 2 previous similar messages [ 6890.288906] Lustre: lustre-MDT0001: Client b677be05-767e-4d66-bf87-657394a294d1 (at 192.168.202.51@tcp) reconnecting [ 6895.261274] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.282856] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6895.295285] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6895.311140] LustreError: 243053:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6895.311523] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6895.325404] LustreError: 243053:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6895.325538] LustreError: 243053:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.151@tcp: cfg command failed: rc = -108 [ 6895.325561] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6895.325561] [ 6895.329321] LustreError: 239331:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6896.111331] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6898.313055] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6902.995786] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6903.199525] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6908.170957] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6908.416850] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6910.906647] Lustre: Permanently deactivating lustre-MDT0001 [ 6910.923676] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6910.927418] Lustre: Skipped 2 previous similar messages [ 6920.856953] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6927.556918] Lustre: server umount lustre-MDT0000 complete [ 6929.900203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6929.910290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6929.936866] LustreError: 240098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6931.441184] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.491951] LustreError: 240079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270496 with bad export cookie 4101066987331085339 [ 6931.499293] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6931.501901] LustreError: 240079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6931.830732] Lustre: server umount lustre-MDT0001 complete [ 6935.681990] Lustre: server umount lustre-OST0000 complete [ 6946.332867] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:35:10 (1789270510) [ 6955.521331] Lustre: server umount lustre-OST0001 complete [ 6961.458974] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 6971.695836] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7003.887143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.103817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7004.118899] Lustre: Skipped 2 previous similar messages [ 7004.129824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7004.238300] Lustre: lustre-MDT0000: new disk, initializing [ 7004.324798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7004.336852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7007.896846] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7016.713697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7016.812540] Lustre: 247598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7016.862078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7016.865542] Lustre: Skipped 1 previous similar message [ 7016.943747] Lustre: lustre-MDT0001: new disk, initializing [ 7017.072575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7017.120271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7017.133505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7021.560266] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7031.103952] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7036.851928] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7043.339144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7043.547823] Lustre: lustre-OST0000: new disk, initializing [ 7043.551865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7043.556444] Lustre: 249563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7043.653433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7049.616618] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7052.363839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7052.375316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7052.469176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7058.438355] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7060.533089] LustreError: 247604:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7068.610612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.866676] Lustre: lustre-OST0001: new disk, initializing [ 7068.870128] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7068.883634] Lustre: 250676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7070.636090] LustreError: 247604:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7070.663022] LustreError: 249571:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7074.667552] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7076.951588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7076.986638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7077.030673] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7080.727118] LustreError: 249571:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7083.517892] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7085.248701] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7087.373440] Lustre: lustre-OST0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 7087.591223] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7087.599441] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.600325] LustreError: Skipped 1 previous similar message [ 7087.616576] Lustre: Skipped 2 previous similar messages [ 7092.208332] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7092.215540] Lustre: Skipped 3 previous similar messages [ 7093.420467] Lustre: server umount lustre-OST0001 complete [ 7097.314455] LustreError: 249549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.337108] LustreError: 249549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7112.684995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.695894] Lustre: Skipped 1 previous similar message [ 7112.707864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.712856] Lustre: Skipped 2 previous similar messages [ 7117.800229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7117.812246] Lustre: Skipped 1 previous similar message [ 7117.991604] Lustre: server umount lustre-OST0000 complete [ 7122.913019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7122.919790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7122.936053] Lustre: Skipped 2 previous similar messages [ 7122.950166] Lustre: Skipped 1 previous similar message [ 7128.509332] Lustre: server umount lustre-MDT0000 complete [ 7132.196074] LustreError: 247590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270697 with bad export cookie 4101066987331088153 [ 7132.209951] LustreError: 247590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7132.211038] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7138.782446] Lustre: server umount lustre-MDT0001 complete [ 7150.092318] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7152.554958] Key type lgssc unregistered [ 7152.871740] LNet: 252707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7152.875203] LNetError: 252707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7152.890802] LNet: Removed LNI 192.168.202.151@tcp [ 7153.720293] Key type .llcrypt unregistered [ 7153.722227] Key type ._llcrypt unregistered [ 7195.174234] Key type ._llcrypt registered [ 7195.181238] Key type .llcrypt registered [ 7195.350451] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7197.291405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7197.338351] alg: No test for adler32 (adler32-zlib) [ 7198.355613] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 7198.544516] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7200.200292] Key type lgssc registered [ 7201.801206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7208.741277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7208.755764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7209.898602] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7209.909260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7210.002730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.468441] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7222.793519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7222.865976] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7222.890754] Lustre: 254914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7223.075408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7227.240403] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7235.934079] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7243.054794] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7250.750353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.863368] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7251.080094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7258.380968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7259.321531] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7267.838172] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7325.171788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.174186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7325.190033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.193392] Lustre: Skipped 1 previous similar message [ 7325.227669] LustreError: Skipped 1 previous similar message [ 7326.878711] Lustre: server umount lustre-OST0000 complete [ 7331.825466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7347.435434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7347.899300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7354.809734] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7362.680159] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7365.288471] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7365.302292] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7365.304759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7406.560374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7406.573683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.582836] LustreError: Skipped 1 previous similar message [ 7406.586048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.608920] Lustre: Skipped 1 previous similar message [ 7406.621830] Lustre: Skipped 2 previous similar messages [ 7408.026412] Lustre: server umount lustre-OST0000 complete [ 7412.704521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7412.710457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.728860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7416.806966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7416.831144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.506073] Lustre: server umount lustre-MDT0000 complete [ 7422.450733] LustreError: 258528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270987 with bad export cookie 7352027635037165568 [ 7422.454890] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7422.469410] LustreError: 258528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7422.801346] Lustre: server umount lustre-MDT0001 complete [ 7434.633783] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7437.174725] Key type lgssc unregistered [ 7437.490806] LNet: 260338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7437.499927] LNetError: 260338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7437.517367] LNet: Removed LNI 192.168.202.151@tcp [ 7438.365816] Key type .llcrypt unregistered [ 7438.374396] Key type ._llcrypt unregistered [ 7463.488972] Key type ._llcrypt registered [ 7463.492740] Key type .llcrypt registered [ 7463.696303] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7465.961991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7465.983148] alg: No test for adler32 (adler32-zlib) [ 7467.129945] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 7467.533933] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7469.223453] Key type lgssc registered [ 7470.636887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7478.881898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7478.913202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.486735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.960946] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7490.019192] LustreError: 261749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.050267] LustreError: 261749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7495.144748] LustreError: 261750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7495.429034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.925587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7501.187972] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7509.243335] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7515.256259] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7523.254400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.657882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.731032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7530.372589] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7539.650282] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7565.793360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7565.801705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.836765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7566.834170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7566.853205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7566.854057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7566.889806] Lustre: Skipped 1 previous similar message [ 7571.219853] Lustre: server umount lustre-OST0000 complete [ 7577.057196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7577.068830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7577.074241] Lustre: Skipped 1 previous similar message [ 7582.177625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7582.187788] Lustre: Skipped 1 previous similar message [ 7587.296752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7587.303274] Lustre: Skipped 1 previous similar message [ 7589.343209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7589.547627] Lustre: server umount lustre-MDT0000 complete [ 7592.417991] LustreError: 263890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7592.430406] LustreError: 263890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7592.990912] LustreError: 261730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271158 with bad export cookie 12653068583496462916 [ 7592.999152] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7593.006731] LustreError: 261730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7593.357690] Lustre: server umount lustre-MDT0001 complete [ 7603.476964] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7606.401748] Key type lgssc unregistered [ 7606.663992] LNet: 266058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7606.674199] LNetError: 266058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7606.684707] LNet: Removed LNI 192.168.202.151@tcp [ 7607.479929] Key type .llcrypt unregistered [ 7607.481110] Key type ._llcrypt unregistered [ 7626.296257] Key type ._llcrypt registered [ 7626.297520] Key type .llcrypt registered [ 7626.390585] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7627.533675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7627.692550] alg: No test for adler32 (adler32-zlib) [ 7628.746141] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 7629.002837] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7630.695658] Key type lgssc registered [ 7631.647102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7639.155228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7639.174614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7640.594334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7644.772639] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7650.784936] LustreError: 267277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7650.800083] LustreError: 267277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7652.527040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.846573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7657.189753] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7665.335420] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7672.306797] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7679.723673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7680.097774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7686.254161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7686.604864] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7693.546719] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7709.161329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.162774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7709.178399] Lustre: Skipped 1 previous similar message [ 7709.179743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7709.182369] Lustre: Skipped 1 previous similar message [ 7713.737936] Lustre: server umount lustre-OST0000 complete [ 7719.400807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7719.410146] Lustre: Skipped 1 previous similar message [ 7719.412810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7719.415083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7724.441585] Lustre: server umount lustre-MDT0000 complete [ 7724.517654] LustreError: 268005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7724.539121] LustreError: 268005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7728.567026] LustreError: 267256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271294 with bad export cookie 14697741342979579382 [ 7728.576755] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7728.928843] Lustre: server umount lustre-MDT0001 complete [ 7739.931492] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7742.653757] Key type lgssc unregistered [ 7742.916579] LNet: 270956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7742.920626] LNetError: 270956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7742.945411] LNet: Removed LNI 192.168.202.151@tcp [ 7743.722609] Key type .llcrypt unregistered [ 7743.727537] Key type ._llcrypt unregistered [ 7768.919651] Key type ._llcrypt registered [ 7768.924425] Key type .llcrypt registered [ 7769.055251] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7770.055432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7770.321649] alg: No test for adler32 (adler32-zlib) [ 7771.415337] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 7771.669698] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7773.399265] Key type lgssc registered [ 7774.551425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7782.363021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7782.391901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.904256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.432473] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7794.149174] LustreError: 272370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.169955] LustreError: 272370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7797.070324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.498242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7801.401710] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7809.326958] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7815.827797] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7823.415314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7823.725405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.869020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7830.105192] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7838.510770] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7867.875851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7867.901516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7867.914925] Lustre: Skipped 1 previous similar message [ 7871.610919] Lustre: server umount lustre-OST0000 complete [ 7878.112516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.134969] Lustre: Skipped 1 previous similar message [ 7878.145024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7883.234086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7883.240097] Lustre: Skipped 2 previous similar messages [ 7888.353737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.358223] Lustre: Skipped 1 previous similar message [ 7889.375651] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7889.656591] Lustre: server umount lustre-MDT0000 complete [ 7893.325960] LustreError: 272349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271458 with bad export cookie 11493486530649467826 [ 7893.335468] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7893.342290] LustreError: 272349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7893.610915] Lustre: server umount lustre-MDT0001 complete [ 7906.021929] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7908.942428] Key type lgssc unregistered [ 7909.288022] LNet: 276676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7909.294390] LNetError: 276676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7909.312188] LNet: Removed LNI 192.168.202.151@tcp [ 7910.303773] Key type .llcrypt unregistered [ 7910.305451] Key type ._llcrypt unregistered [ 7929.375453] Key type ._llcrypt registered [ 7929.376872] Key type .llcrypt registered [ 7929.484990] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7930.883399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7930.940582] alg: No test for adler32 (adler32-zlib) [ 7931.973769] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 7932.259814] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7933.951246] Key type lgssc registered [ 7935.611491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7943.750699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7943.779364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.626800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7949.445731] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7955.941901] LustreError: 277911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7955.959468] LustreError: 277911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7957.917874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7958.286636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7962.893874] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7970.795271] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7977.095483] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7984.133979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7984.356835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.054525] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7991.557565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8000.359823] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8017.378078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8017.388851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.405668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8018.400296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8018.422027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.423562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8022.266147] Lustre: server umount lustre-OST0000 complete [ 8028.643048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.649176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8028.664334] Lustre: Skipped 1 previous similar message [ 8028.684729] Lustre: Skipped 2 previous similar messages [ 8033.072399] Lustre: server umount lustre-MDT0000 complete [ 8033.762844] LustreError: 277905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8033.779561] LustreError: 277905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8037.359498] LustreError: 277877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271602 with bad export cookie 7504079391463375650 [ 8037.364950] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8037.374944] LustreError: 277877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8037.643402] Lustre: server umount lustre-MDT0001 complete [ 8051.606592] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8054.833568] Key type lgssc unregistered [ 8055.154583] LNet: 281613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8055.178888] LNetError: 281613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8055.207477] LNet: Removed LNI 192.168.202.151@tcp [ 8056.239257] Key type .llcrypt unregistered [ 8056.250900] Key type ._llcrypt unregistered [ 8070.296174] Key type ._llcrypt registered [ 8070.303641] Key type .llcrypt registered [ 8070.435485] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8087.748835] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8107.260658] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8109.572379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8109.665859] alg: No test for adler32 (adler32-zlib) [ 8110.806732] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 8111.026993] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8112.744318] Key type lgssc registered [ 8114.101529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8151.543914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8151.571779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.854682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8152.905375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8153.035762] Lustre: lustre-MDT0000: new disk, initializing [ 8153.193831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.214362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8158.152700] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8170.024727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.259938] Lustre: 285180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8170.491890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8170.501723] Lustre: Skipped 1 previous similar message [ 8170.609466] Lustre: lustre-MDT0001: new disk, initializing [ 8170.695618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8170.725353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8170.747171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8176.243218] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8187.319507] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8193.590926] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8200.292489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8200.482783] Lustre: lustre-OST0000: new disk, initializing [ 8200.486904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8200.494941] Lustre: 287152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8200.585705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8206.958877] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8207.907935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8207.913526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8208.064593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8216.248275] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8220.411857] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8220.617286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8225.140861] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8225.296730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8228.320552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8228.328593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8228.350503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8228.832301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8228.835537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8233.197573] Lustre: server umount lustre-OST0000 complete [ 8237.541014] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8237.552735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.577810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8237.582176] Lustre: Skipped 1 previous similar message [ 8239.077275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8239.082920] Lustre: Skipped 1 previous similar message [ 8244.192864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8244.197215] Lustre: Skipped 1 previous similar message [ 8249.314318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8249.322197] Lustre: Skipped 1 previous similar message [ 8251.359744] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8251.599463] Lustre: server umount lustre-MDT0000 complete [ 8255.225036] LustreError: 285173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271820 with bad export cookie 7642483221942910715 [ 8255.229776] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8255.237417] LustreError: 285173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8255.593306] Lustre: server umount lustre-MDT0001 complete [ 8264.138322] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:57:07 (1789271827) [ 8270.406193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8270.841661] LustreError: 289563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.857589] LustreError: 289563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8270.915307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8275.941658] LustreError: 289564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8276.071538] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8281.069863] LustreError: 289563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8284.713485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8285.035175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8288.981711] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8296.595916] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8301.919575] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8308.848141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8309.238882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8316.249551] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8324.007133] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8335.841400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8335.855839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8335.869390] Lustre: Skipped 1 previous similar message [ 8335.881591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8341.093788] Lustre: server umount lustre-OST0000 complete [ 8346.592701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8346.602594] LustreError: Skipped 1 previous similar message [ 8346.608698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8346.638037] Lustre: Skipped 1 previous similar message [ 8352.224959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8352.247333] Lustre: Skipped 6 previous similar messages [ 8359.391310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8359.665527] Lustre: server umount lustre-MDT0000 complete [ 8362.465123] LustreError: 289564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8362.486808] LustreError: 289564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8363.291861] LustreError: 289545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271928 with bad export cookie 7642483221942911597 [ 8363.303438] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8363.313897] LustreError: 289545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8363.649289] Lustre: server umount lustre-MDT0001 complete [ 8375.517608] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8378.042709] Key type lgssc unregistered [ 8378.280448] LNet: 293099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8378.290123] LNetError: 293099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8378.315942] LNet: Removed LNI 192.168.202.151@tcp [ 8379.376387] Key type .llcrypt unregistered [ 8379.378353] Key type ._llcrypt unregistered [ 8405.406489] Key type ._llcrypt registered [ 8405.409874] Key type .llcrypt registered [ 8405.537248] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8423.168725] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8424.920344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8424.956029] alg: No test for adler32 (adler32-zlib) [ 8426.177081] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 8426.442940] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8428.138293] Key type lgssc registered [ 8429.504176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8480.211918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8480.232463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8481.622128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8481.670215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8481.726104] Lustre: lustre-MDT0000: new disk, initializing [ 8481.777300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8481.792507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8485.952877] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8496.962598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8497.080784] Lustre: 296928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8497.144140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8497.148592] Lustre: Skipped 1 previous similar message [ 8497.202564] Lustre: lustre-MDT0001: new disk, initializing [ 8497.285202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8497.324909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8497.337782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8501.998276] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8511.258799] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8516.961967] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8524.262057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8524.603570] Lustre: lustre-OST2710: new disk, initializing [ 8524.612474] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8524.625629] Lustre: 298907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8524.765959] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8531.525154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8531.541421] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8531.647438] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8532.648869] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8545.169267] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8553.801880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8554.041499] Lustre: lustre-OST03e8: new disk, initializing [ 8554.045739] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8554.051259] Lustre: 299972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8554.193322] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8560.346052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8560.378942] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8560.473595] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8562.249596] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8571.106696] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8580.178471] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8580.472884] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8585.901031] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8586.388791] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8591.657065] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8591.974902] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8596.738267] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8597.013650] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8601.583475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8601.593986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8601.604521] Lustre: Skipped 2 previous similar messages [ 8601.618728] Lustre: Skipped 2 previous similar messages [ 8606.695472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8606.711345] Lustre: Skipped 3 previous similar messages [ 8607.345940] Lustre: server umount lustre-MDT0000 complete [ 8612.462490] LustreError: 296919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272177 with bad export cookie 1595000967326274495 [ 8612.467726] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8612.478882] LustreError: 296919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8612.927886] Lustre: server umount lustre-MDT0001 complete [ 8627.887735] Lustre: server umount lustre-OST2710 complete [ 8643.935424] Lustre: server umount lustre-OST03e8 complete [ 8669.054050] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8684.549660] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8731.783754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.951696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8731.979985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8732.032861] Lustre: lustre-MDT0000: new disk, initializing [ 8732.095954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8732.110338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8736.096474] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8748.579531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.770656] Lustre: 305258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8748.816398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8748.820021] Lustre: Skipped 1 previous similar message [ 8748.895550] Lustre: lustre-MDT0001: new disk, initializing [ 8748.995875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8749.052384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8749.070507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8754.862450] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8766.050510] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8772.289272] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8779.640915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8780.072271] Lustre: lustre-OST0000: new disk, initializing [ 8780.077880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8780.088933] Lustre: 307228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8780.204154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8787.552303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8787.563455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8787.638244] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8787.742665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8798.369742] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8803.291549] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8803.490828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8808.483645] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8808.763794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8813.536518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8813.549271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.554301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8813.562169] Lustre: Skipped 2 previous similar messages [ 8813.570773] Lustre: Skipped 1 previous similar message [ 8817.638723] Lustre: server umount lustre-OST0000 complete [ 8823.777628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8823.803846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8823.810183] Lustre: Skipped 1 previous similar message [ 8828.898291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8837.090323] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8837.486819] Lustre: server umount lustre-MDT0000 complete [ 8839.140172] LustreError: 305268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8839.160880] LustreError: 305268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8841.656290] LustreError: 307227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272407 with bad export cookie 1595000967326276175 [ 8841.664271] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8841.666542] LustreError: 307227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8842.115647] Lustre: server umount lustre-MDT0001 complete [ 8853.156656] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:06:56 (1789272416) [ 8855.303945] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8857.598810] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:07:00 (1789272420) [ 8861.972547] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8894.981577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8895.290934] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8895.313641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8895.533876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8901.467472] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8913.277221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8913.450736] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8913.499501] Lustre: 311087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8921.228689] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8944.516939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8944.878923] Lustre: Denying initial registration attempt from nid 192.168.202.151@tcp, specified as failover [ 8944.891666] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8944.897382] LustreError: 312544:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 8944.910136] LustreError: 312544:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8944.918949] LustreError: 312544:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8945.046413] Lustre: server umount lustre-OST0000 complete [ 8945.051448] LustreError: 312544:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8954.338394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8954.342747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8954.358624] Lustre: Skipped 2 previous similar messages [ 8954.373763] Lustre: Skipped 4 previous similar messages [ 8959.017749] Lustre: server umount lustre-MDT0000 complete [ 8959.457023] LustreError: 311099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.487934] LustreError: 311099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8964.577306] LustreError: 312179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8964.597204] LustreError: 312179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8964.830092] LustreError: 311080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272530 with bad export cookie 1595000967326277057 [ 8964.836451] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8964.836599] LustreError: 311080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8965.142948] Lustre: server umount lustre-MDT0001 complete [ 8978.504576] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8981.381605] Key type lgssc unregistered [ 8981.670032] LNet: 313666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8981.677046] LNetError: 313666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8982.702052] LNet: Removed LNI 192.168.202.151@tcp [ 8983.963324] Key type .llcrypt unregistered [ 8983.974800] Key type ._llcrypt unregistered [ 8997.244736] Key type ._llcrypt registered [ 8997.248411] Key type .llcrypt registered [ 8997.354489] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8999.119925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8999.150516] alg: No test for adler32 (adler32-zlib) [ 9000.275531] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 9000.658229] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9002.399411] Key type lgssc registered [ 9003.482500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9054.990578] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9066.250305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9066.309061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9067.784588] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9067.834087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9068.048298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9073.328916] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9085.431827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.522459] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9085.551934] Lustre: 316427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9085.831707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9092.303405] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9104.465249] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9111.251624] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9133.569378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9133.840755] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9134.150127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9143.346312] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9153.226199] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9159.150033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.162080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9159.192973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9159.194723] LustreError: Skipped 1 previous similar message [ 9159.222471] Lustre: Skipped 1 previous similar message [ 9162.064897] Lustre: server umount lustre-OST0000 complete [ 9167.859766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9167.876358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.896981] Lustre: Skipped 1 previous similar message [ 9167.902572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9169.384354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9169.399444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9173.354675] Lustre: server umount lustre-MDT0000 complete [ 9177.818375] LustreError: 316420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272743 with bad export cookie 12305707560397727235 [ 9177.823364] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9177.832896] LustreError: 316420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9178.381153] Lustre: server umount lustre-MDT0001 complete [ 9191.173425] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9194.693134] Key type lgssc unregistered [ 9195.022554] LNet: 320002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9195.041677] LNetError: 320002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9195.077780] LNet: Removed LNI 192.168.202.151@tcp [ 9196.425892] Key type .llcrypt unregistered [ 9196.427809] Key type ._llcrypt unregistered [ 9230.774605] Key type ._llcrypt registered [ 9230.777382] Key type .llcrypt registered [ 9230.881808] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9231.864794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9231.903251] alg: No test for adler32 (adler32-zlib) [ 9233.089733] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 9233.387487] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9235.079196] Key type lgssc registered [ 9236.501609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9245.384651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9245.417594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9247.123400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9252.345322] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9257.446016] LustreError: 321417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9257.470493] LustreError: 321417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9261.679741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.189614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9267.531926] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9276.186972] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9284.143920] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9293.927144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9294.456335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9301.441977] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9309.945374] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9331.466346] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 9332.713617] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.719939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9332.743095] Lustre: Skipped 1 previous similar message [ 9332.769899] Lustre: Skipped 2 previous similar messages [ 9336.246398] Lustre: server umount lustre-MDT0000 complete [ 9336.561805] LustreError: 321412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9337.836803] LustreError: 323907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9337.861322] LustreError: 323907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9341.686616] LustreError: 321413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9341.918681] LustreError: 321397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272907 with bad export cookie 790372973042055642 [ 9341.932859] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9341.939439] LustreError: 321397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9342.946325] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.955117] Lustre: Skipped 1 previous similar message [ 9342.960397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9346.791731] LustreError: 321412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9346.838268] LustreError: 321412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9346.843908] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 9351.923157] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 9351.930893] Lustre: Skipped 1 previous similar message [ 9356.255522] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9356.611850] Lustre: server umount lustre-MDT0001 complete [ 9364.947134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9377.664595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9388.455572] LustreError: 323248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9388.475636] LustreError: 323248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9388.549591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9388.606516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9394.075761] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9403.957673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.599806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9406.646429] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9410.474862] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9421.776743] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9425.736799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9433.823636] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9435.838287] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9442.786510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.789268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9442.807628] Lustre: Skipped 1 previous similar message [ 9442.825014] Lustre: Skipped 2 previous similar messages [ 9456.095166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9456.553736] Lustre: server umount lustre-OST0000 complete [ 9463.286160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9463.299090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9463.312173] Lustre: Skipped 5 previous similar messages [ 9467.665873] Lustre: server umount lustre-MDT0000 complete [ 9468.388671] LustreError: 325034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9468.416455] LustreError: 325034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9472.621595] LustreError: 321397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273038 with bad export cookie 790372973042063062 [ 9472.633920] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9473.112748] Lustre: server umount lustre-MDT0001 complete [ 9488.501720] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9492.256441] Key type lgssc unregistered [ 9492.809351] LNet: 327709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9492.817494] LNetError: 327709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9492.840440] LNet: Removed LNI 192.168.202.151@tcp [ 9494.324468] Key type .llcrypt unregistered [ 9494.326351] Key type ._llcrypt unregistered [ 9526.435981] Key type ._llcrypt registered [ 9526.438963] Key type .llcrypt registered [ 9526.609658] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9528.038773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9528.291470] alg: No test for adler32 (adler32-zlib) [ 9529.385655] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 9529.628026] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9531.303181] Key type lgssc registered [ 9533.043961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9542.727455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9542.793843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.806081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.544192] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9554.918424] LustreError: 329122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9554.932729] LustreError: 329122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9560.034615] LustreError: 329123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9563.335275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9564.203233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9569.534769] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9579.918481] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9588.071129] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9594.848620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9594.856715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9594.862232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9594.892325] Lustre: Skipped 1 previous similar message [ 9598.033952] Lustre: server umount lustre-MDT0000 complete [ 9599.971615] LustreError: 329119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9599.994779] LustreError: 329119:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9602.420765] LustreError: 329104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273167 with bad export cookie 14029952353309602052 [ 9602.425650] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9602.433976] LustreError: 329104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9603.085788] Lustre: server umount lustre-MDT0001 complete [ 9622.206404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.743474] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9622.774750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9622.940592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9628.477180] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9641.657768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.863510] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9641.950558] Lustre: 331713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9642.229802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9647.532373] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9656.481948] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9664.087614] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9676.609696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9676.957731] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9676.994822] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9677.001952] LustreError: 333601:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9677.006939] LustreError: 333601:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9677.011241] LustreError: 333601:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9677.096094] Lustre: server umount lustre-OST0000 complete [ 9677.100230] LustreError: 333601:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9688.696527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9688.913955] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9689.156737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9697.014761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9697.831757] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9707.739167] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9718.050251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9718.161273] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9718.239780] Lustre: lustre-OST0001: new disk, initializing [ 9718.245810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9718.254174] Lustre: 335142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9718.347547] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9726.175769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9726.199085] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9726.379537] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9727.504624] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9741.746251] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9746.913736] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9746.917208] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9746.925694] LustreError: Skipped 1 previous similar message [ 9746.942082] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9746.943816] Lustre: Skipped 1 previous similar message [ 9746.956486] Lustre: Skipped 2 previous similar messages [ 9751.987834] Lustre: server umount lustre-OST0001 complete [ 9752.045919] LustreError: 334068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9752.073208] LustreError: 334068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9757.155511] LustreError: 334068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.158635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9757.159751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9757.166882] LustreError: 334068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9757.185944] Lustre: Skipped 1 previous similar message [ 9762.285407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9762.292426] Lustre: Skipped 2 previous similar messages [ 9763.556437] Lustre: server umount lustre-OST0000 complete [ 9767.394529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.414747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9767.422863] Lustre: Skipped 1 previous similar message [ 9767.437581] Lustre: Skipped 1 previous similar message [ 9773.765434] Lustre: server umount lustre-MDT0000 complete [ 9777.649409] LustreError: 331726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9777.662123] LustreError: 331726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9777.955025] LustreError: 332456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273343 with bad export cookie 14029952353309602605 [ 9777.967902] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9777.969611] LustreError: 332456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9778.349028] Lustre: server umount lustre-MDT0001 complete [ 9793.517177] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9797.563223] Key type lgssc unregistered [ 9798.092051] LNet: 337100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9798.100961] LNetError: 337100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9798.128958] LNet: Removed LNI 192.168.202.151@tcp [ 9799.376749] Key type .llcrypt unregistered [ 9799.380271] Key type ._llcrypt unregistered [ 9888.038213] Key type ._llcrypt registered [ 9888.040591] Key type .llcrypt registered [ 9888.153164] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 9906.063772] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9907.550994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9907.593791] alg: No test for adler32 (adler32-zlib) [ 9908.785702] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 9909.115429] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9910.823190] Key type lgssc registered [ 9912.010503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9949.808661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9949.880046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9951.205374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9951.300109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9951.419529] Lustre: lustre-MDT0000: new disk, initializing [ 9951.576430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9951.609753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9957.739279] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9971.476630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.678840] Lustre: 342037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9971.715777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9971.718492] Lustre: Skipped 1 previous similar message [ 9971.769778] Lustre: lustre-MDT0001: new disk, initializing [ 9971.842474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9971.901337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9971.911761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9977.456392] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9989.581925] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9998.948082] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10012.047664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10012.483961] Lustre: lustre-OST0000: new disk, initializing [10012.489381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10012.501838] Lustre: 344008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10012.630470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10019.282231] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10020.520994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10020.534317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10020.704044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10029.469696] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10036.154311] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10036.450390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10041.437110] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10041.692807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10046.432365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10046.433630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10046.444431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10046.453446] LustreError: Skipped 1 previous similar message [10046.514739] Lustre: Skipped 1 previous similar message [10049.822958] Lustre: server umount lustre-OST0000 complete [10054.112410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10054.120064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10054.135621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10054.147376] Lustre: Skipped 1 previous similar message [10056.673887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10056.678256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10056.709187] Lustre: Skipped 1 previous similar message [10060.410723] Lustre: server umount lustre-MDT0000 complete [10065.037791] LustreError: 342027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273630 with bad export cookie 3172560174329978733 [10065.040389] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10065.048378] LustreError: 342027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10065.278535] Lustre: server umount lustre-MDT0001 complete [10076.950415] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:27:20 (1789273640) [10086.333674] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:27:29 (1789273649) [10097.287038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10097.863122] LustreError: 346765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10097.883589] LustreError: 346765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10097.965574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10103.264332] LustreError: 346766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10103.665100] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10108.391531] LustreError: 346765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.516419] LustreError: 346766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10116.098823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.785888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10120.764635] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10130.239097] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10137.438587] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10145.844459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10146.516312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10154.363790] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10166.259228] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10175.901865] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10177.929140] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10180.120603] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10182.248930] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10184.836313] Lustre: Failing over lustre-MDT0000 [10184.993803] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [10185.575680] Lustre: server umount lustre-MDT0000 complete [10188.257141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10188.262248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10188.276398] LustreError: 346766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10199.012514] LustreError: 347494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10199.041909] LustreError: 347494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10205.154469] Lustre: 340257:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789273754/real 1789273754] req@00000000589b232d x1876189222351104/t0(0) o400->MGC192.168.202.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789273770 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10205.210865] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10208.059084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.393431] LustreError: 346761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10215.412266] LustreError: 346761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [10216.418505] LustreError: 340255:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006a1d72bf x1876189222356992/t0(0) o250->MGC192.168.202.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10216.693188] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (not set up) [10216.918362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10217.052526] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10221.824472] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10222.072719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10222.129878] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10222.227866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10222.540703] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10237.633338] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10240.278703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10246.431669] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10249.247915] Lustre: Failing over lustre-MDT0000 [10249.608660] Lustre: server umount lustre-MDT0000 complete [10252.585949] LustreError: 346761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10252.612867] LustreError: 346761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10252.771709] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10252.776330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10252.798269] Lustre: Skipped 2 previous similar messages [10269.151384] Lustre: 340259:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789273818/real 1789273818] req@00000000019c6d52 x1876189222375936/t0(0) o400->MGC192.168.202.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789273834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10269.214238] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10271.387697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10280.425774] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2c0732ab6b7d6a10 [10280.476547] Lustre: MGC192.168.202.151@tcp: Connection restored to 0@lo (at 0@lo) [10280.479066] Lustre: Skipped 2 previous similar messages [10280.899899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10281.004503] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10283.756243] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10285.741628] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10286.086511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10286.164770] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10286.287026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10297.579366] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10300.025946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10311.649578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10311.656967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10311.675866] Lustre: Skipped 4 previous similar messages [10311.706735] Lustre: Skipped 2 previous similar messages [10315.261352] Lustre: server umount lustre-MDT0000 complete [10316.772783] LustreError: 346765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10316.807858] LustreError: 346765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 31 previous similar messages [10319.719662] LustreError: 346747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273885 with bad export cookie 3172560174329981456 [10319.753074] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10320.163579] Lustre: server umount lustre-MDT0001 complete [10335.271919] Lustre: server umount lustre-OST0000 complete [10352.931489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.465525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10358.445017] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10370.424653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10375.993528] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10385.030363] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10392.923428] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10400.196574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10408.843936] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10419.014648] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10423.115536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10428.388335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10428.394761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10428.415812] Lustre: Skipped 1 previous similar message [10428.422414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10428.426896] Lustre: Skipped 1 previous similar message [10430.088903] Lustre: server umount lustre-OST0000 complete [10437.601615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10440.706335] Lustre: server umount lustre-MDT0000 complete [10444.879517] LustreError: 353098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274010 with bad export cookie 3172560174329982198 [10444.900290] LustreError: 353098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10444.904464] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10445.270993] Lustre: server umount lustre-MDT0001 complete [10459.062826] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [10462.501508] Key type lgssc unregistered [10462.941267] LNet: 356652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10462.964043] LNetError: 356652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10463.014363] LNet: Removed LNI 192.168.202.151@tcp [10464.650095] Key type .llcrypt unregistered [10464.659603] Key type ._llcrypt unregistered [10502.158302] Key type ._llcrypt registered [10502.160728] Key type .llcrypt registered [10502.285205] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10503.275382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10503.397613] alg: No test for adler32 (adler32-zlib) [10504.779797] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [10505.124751] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [10506.847157] Key type lgssc registered [10508.681791] Lustre: Echo OBD driver; http://www.lustre.org/ [10519.798417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10519.812249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10521.319982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10526.555775] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10530.786581] LustreError: 358167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.823952] LustreError: 358167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10535.911411] LustreError: 358168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10536.196774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10536.428924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10540.197758] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10549.487682] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10556.761354] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10563.567440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10563.898328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10565.962572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10570.281635] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10579.423684] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10597.859594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10597.863829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10597.880605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10601.953542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10601.957295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10601.970299] Lustre: Skipped 1 previous similar message [10604.230310] Lustre: server umount lustre-MDT0000 complete [10607.072775] LustreError: 360669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10607.100730] LustreError: 360669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10609.999513] LustreError: 358149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274175 with bad export cookie 10103344842699232947 [10610.010833] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10610.013132] LustreError: 358149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10610.458461] Lustre: server umount lustre-MDT0001 complete [10625.819296] Lustre: server umount lustre-OST0000 complete [10639.816027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10640.391089] LustreError: 361901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10640.513401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10645.751911] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10645.990090] LustreError: 361902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10655.201136] LustreError: 361901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10655.230027] LustreError: 361901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10655.974342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10656.693804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10663.483422] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10672.282699] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10678.720561] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10685.462268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10685.730076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10691.728124] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10699.852768] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10710.340771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10725.868189] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10725.876741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10725.886172] Lustre: Skipped 3 previous similar messages [10725.900327] Lustre: Skipped 2 previous similar messages [10730.986901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10730.994445] Lustre: Skipped 2 previous similar messages [10731.205879] Lustre: server umount lustre-MDT0000 complete [10735.177729] LustreError: 361883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274300 with bad export cookie 10103344842699234060 [10735.182406] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10735.199519] LustreError: 361883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10735.699111] Lustre: server umount lustre-MDT0001 complete [10750.034627] Lustre: server umount lustre-OST0000 complete [10771.004723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.518676] LustreError: 365834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10771.617251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10776.662937] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10785.577401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10791.775750] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10799.195688] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10806.924806] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10816.675675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10817.283435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10817.295329] Lustre: Skipped 1 previous similar message [10818.380041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10824.905649] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10834.016592] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10844.136331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10844.145944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10844.157383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10846.198059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10849.542900] Lustre: server umount lustre-OST0000 complete [10856.421901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10856.426237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10856.435038] Lustre: Skipped 2 previous similar messages [10856.452195] Lustre: Skipped 3 previous similar messages [10869.215392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10870.754629] LustreError: 365831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10870.784917] LustreError: 365831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10871.544048] Lustre: server umount lustre-MDT0000 complete [10876.104202] LustreError: 365815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274441 with bad export cookie 10103344842699235082 [10876.106370] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10876.122627] LustreError: 365815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10876.682757] Lustre: server umount lustre-MDT0001 complete [10892.686969] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [10895.725242] Key type lgssc unregistered [10896.145810] LNet: 369371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10896.159218] LNetError: 369371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10896.192027] LNet: Removed LNI 192.168.202.151@tcp [10897.180467] Key type .llcrypt unregistered [10897.183278] Key type ._llcrypt unregistered [10927.039946] Key type ._llcrypt registered [10927.041973] Key type .llcrypt registered [10927.199891] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10928.659297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10928.703339] alg: No test for adler32 (adler32-zlib) [10929.939851] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [10930.212269] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [10931.903185] Key type lgssc registered [10933.030700] Lustre: Echo OBD driver; http://www.lustre.org/ [10941.272224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10941.286896] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10941.292161] LustreError: 370808:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10941.320435] LustreError: 370808:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10941.340831] LustreError: 370808:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10941.351600] LustreError: 370808:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10941.360889] LustreError: 370808:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10949.880773] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10949.892765] LustreError: 371205:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10949.927732] LustreError: 371205:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10949.937576] LustreError: 371205:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10949.945261] LustreError: 371205:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10949.952743] LustreError: 371205:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10971.727824] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [10974.357505] Key type lgssc unregistered [10974.592490] LNet: 372210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10974.602188] LNetError: 372210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10974.622625] LNet: Removed LNI 192.168.202.151@tcp [10975.669439] Key type .llcrypt unregistered [10975.670785] Key type ._llcrypt unregistered [10998.831848] Key type ._llcrypt registered [10998.837106] Key type .llcrypt registered [10999.043376] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [11017.714620] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11018.698732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11018.839994] alg: No test for adler32 (adler32-zlib) [11019.993921] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [11020.240327] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11021.984712] Key type lgssc registered [11023.335335] Lustre: Echo OBD driver; http://www.lustre.org/ [11059.524317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11059.554288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11060.866713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11060.905077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11061.014184] Lustre: lustre-MDT0000: new disk, initializing [11061.093437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11061.112279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11065.849785] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11076.947595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.027179] Lustre: 375363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11077.072299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11077.075319] Lustre: Skipped 1 previous similar message [11077.130446] Lustre: lustre-MDT0001: new disk, initializing [11077.187477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11077.215141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11077.229376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11082.849919] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11094.368292] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11101.544832] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11108.530656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11108.791515] Lustre: lustre-OST0000: new disk, initializing [11108.795641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11108.806274] Lustre: 377333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11108.899616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11115.660249] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11118.168757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11118.188449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11118.322100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11125.255235] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11129.992478] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11130.256981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11135.510358] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11135.868623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11139.046062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11139.049148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11139.071085] Lustre: Skipped 1 previous similar message [11139.071483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11139.087033] Lustre: Skipped 1 previous similar message [11144.168899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11144.184718] Lustre: Skipped 1 previous similar message [11144.773279] Lustre: server umount lustre-OST0000 complete [11153.889929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11153.907442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11153.945230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11156.827712] Lustre: server umount lustre-MDT0000 complete [11160.555493] LustreError: 375356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274726 with bad export cookie 15345691661050338645 [11160.564525] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11160.573395] LustreError: 375356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11161.131905] Lustre: server umount lustre-MDT0001 complete [11171.196565] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11173.251857] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:45:36 (1789274736) [11180.728703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.307146] LustreError: 379794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11181.357763] LustreError: 379794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11181.480744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11186.657830] LustreError: 379795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11187.087724] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11190.758558] LustreError: 379794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11195.876389] LustreError: 379795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11197.304995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11197.851263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11204.194532] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11212.695902] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11219.263573] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11226.677860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11227.195408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11234.711549] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11247.544093] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11258.989251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11259.252481] Lustre: lustre-OST0001: new disk, initializing [11259.257553] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11259.269551] Lustre: 382632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11259.420547] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11266.166961] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11267.798025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11267.820215] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11267.880890] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11276.397335] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11283.432801] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11283.433234] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11283.447924] LustreError: Skipped 1 previous similar message [11283.459557] Lustre: Skipped 1 previous similar message [11283.466775] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11283.486983] Lustre: Skipped 3 previous similar messages [11287.240284] Lustre: server umount lustre-OST0001 complete [11288.549779] LustreError: 381617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.582460] LustreError: 381617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11298.788653] LustreError: 381618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11298.822860] LustreError: 381618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11308.003042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11308.013441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11308.027613] Lustre: Skipped 1 previous similar message [11308.033986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11308.041085] Lustre: Skipped 1 previous similar message [11312.762399] Lustre: server umount lustre-OST0000 complete [11319.266373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11319.269133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11319.288931] Lustre: Skipped 2 previous similar messages [11319.294507] Lustre: Skipped 2 previous similar messages [11324.012022] Lustre: server umount lustre-MDT0000 complete [11324.384346] LustreError: 379789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11324.390198] LustreError: 379789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11328.852197] LustreError: 379775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274894 with bad export cookie 15345691661050339527 [11328.861633] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11328.865235] LustreError: 379775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11343.327372] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11343.792441] Lustre: server umount lustre-MDT0001 complete [11356.105736] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11358.821501] Key type lgssc unregistered [11359.118805] LNet: 384622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11359.141209] LNetError: 384622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11359.157987] LNet: Removed LNI 192.168.202.151@tcp [11360.150408] Key type .llcrypt unregistered [11360.152043] Key type ._llcrypt unregistered [11419.106430] Key type ._llcrypt registered [11419.108858] Key type .llcrypt registered [11419.207083] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11420.155114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11420.181597] alg: No test for adler32 (adler32-zlib) [11421.458893] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [11421.781442] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11423.543166] Key type lgssc registered [11425.013927] Lustre: Echo OBD driver; http://www.lustre.org/ [11433.936833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11433.965250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11435.446691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11435.504497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11435.787547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11443.212215] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11456.148073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11456.330461] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11456.378770] Lustre: 387419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11456.659121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11461.947822] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11471.898472] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11480.950488] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11486.747339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11486.782798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11486.785730] Lustre: Skipped 1 previous similar message [11489.267724] Lustre: server umount lustre-MDT0000 complete [11494.358780] LustreError: 387413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275059 with bad export cookie 16498348132503388071 [11494.383526] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11494.386483] LustreError: 387413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11494.875258] Lustre: server umount lustre-MDT0001 complete [11508.787089] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:51:11 (1789275071) [11517.661611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11517.946370] LustreError: 389934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11517.969110] LustreError: 389934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11518.049586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11523.043244] LustreError: 389935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.073931] LustreError: 389935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11524.028871] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11528.162655] LustreError: 389934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.284454] LustreError: 389935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.480193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11533.985899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11540.348587] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11550.555197] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11557.674477] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11566.130573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11566.337049] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11566.578708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11573.109546] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11582.626080] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11590.531720] Lustre: Permanently deactivating lustre-OST0000 [11590.551346] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11597.365264] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11598.585929] LustreError: 392875:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11598.597275] LustreError: 392875:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11598.607120] LustreError: 392875:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11603.133950] LustreError: 392923:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11603.141367] LustreError: 392923:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11611.572204] Lustre: server umount lustre-OST0000 complete [11614.841858] LustreError: 393174:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11614.858175] LustreError: 393174:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11619.811572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11619.822165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11619.823994] Lustre: Skipped 2 previous similar messages [11619.843019] Lustre: Skipped 1 previous similar message [11624.929925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11624.941958] Lustre: Skipped 1 previous similar message [11627.096886] Lustre: server umount lustre-MDT0000 complete [11630.066808] LustreError: 389931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11630.089072] LustreError: 389931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11630.978957] LustreError: 389916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275196 with bad export cookie 16498348132503388652 [11630.986598] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11630.998795] LustreError: 389916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11641.557260] Lustre: server umount lustre-MDT0001 complete [11652.355871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11657.017804] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11660.478500] LustreError: 394333:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11667.111683] Lustre: 394672:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.151@tcp [11669.440345] Lustre: server umount MGS complete [11681.505974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11682.039065] LustreError: 395397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11682.067348] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11682.077580] Lustre: Skipped 1 previous similar message [11682.137642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11687.295775] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11696.405663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11696.717705] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11701.934418] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11709.475511] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11715.540188] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11717.450438] Lustre: Permanently reactivating lustre-OST0000 [11717.495749] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11717.508735] Lustre: Skipped 2 previous similar messages [11744.736837] LustreError: 397145:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11755.636161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11756.366600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11756.391611] Lustre: Skipped 1 previous similar message [11765.977377] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11775.922608] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11785.186826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11785.202948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11785.229385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11790.344958] Lustre: server umount lustre-OST0000 complete [11796.452060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11796.462645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11796.467854] Lustre: Skipped 2 previous similar messages [11796.483623] Lustre: Skipped 3 previous similar messages [11801.518260] Lustre: server umount lustre-MDT0000 complete [11801.569270] LustreError: 395397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.596765] LustreError: 395397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11805.485407] LustreError: 395378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275370 with bad export cookie 16498348132503389702 [11805.485880] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11805.504301] LustreError: 395378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11805.957365] Lustre: server umount lustre-MDT0001 complete [11819.292253] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11823.157457] Key type lgssc unregistered [11823.576950] LNet: 399714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11823.587254] LNetError: 399714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11824.619801] LNet: Removed LNI 192.168.202.151@tcp [11825.829992] Key type .llcrypt unregistered [11825.831920] Key type ._llcrypt unregistered [11856.755818] Key type ._llcrypt registered [11856.760825] Key type .llcrypt registered [11856.886823] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [11876.548366] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11877.478477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11877.552481] alg: No test for adler32 (adler32-zlib) [11878.725243] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [11879.061269] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11880.736580] Key type lgssc registered [11882.289479] Lustre: Echo OBD driver; http://www.lustre.org/ [11921.022918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11921.040602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11922.362833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11922.402201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11922.539480] Lustre: lustre-MDT0000: new disk, initializing [11922.646837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11922.670544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11927.941184] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11942.125888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11942.285285] Lustre: 402872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11942.370594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11942.375307] Lustre: Skipped 1 previous similar message [11942.488091] Lustre: lustre-MDT0001: new disk, initializing [11942.646419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11942.697863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11942.720559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11947.796818] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11958.196442] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11964.819524] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11972.933384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11973.380524] Lustre: lustre-OST0000: new disk, initializing [11973.384472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11973.397703] Lustre: 404843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11973.552496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11981.101947] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11981.433225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11981.443346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11981.577625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11991.865769] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11998.433216] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11998.787415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12005.165906] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12005.347866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12012.518027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12012.522176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.527406] LustreError: Skipped 1 previous similar message [12012.547373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12012.555471] Lustre: Skipped 1 previous similar message [12013.804401] Lustre: server umount lustre-OST0000 complete [12019.685082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12019.693407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12019.709238] Lustre: Skipped 1 previous similar message [12019.715306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12022.753221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12022.761227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12024.367980] Lustre: server umount lustre-MDT0000 complete [12027.995500] LustreError: 402865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275593 with bad export cookie 5269387966120115106 [12028.001931] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12028.009039] LustreError: 402865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12028.354644] Lustre: server umount lustre-MDT0001 complete [12037.457867] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12039.338886] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:00:02 (1789275602) [12046.976618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12047.555751] LustreError: 407304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12047.578718] LustreError: 407304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12047.699872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12052.499753] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12052.965601] LustreError: 407305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12058.093088] LustreError: 407304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.604315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12061.929836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12066.809592] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12076.296102] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12083.793915] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12093.570632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.450763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12106.026406] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12117.338572] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12120.453326] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12123.501103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12132.319978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12132.326173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12132.339559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12132.349652] Lustre: Skipped 1 previous similar message [12135.387900] Lustre: server umount lustre-OST0000 complete [12144.096718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12144.116158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.124161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12144.144043] Lustre: Skipped 2 previous similar messages [12144.167855] Lustre: Skipped 3 previous similar messages [12145.842636] Lustre: server umount lustre-MDT0000 complete [12149.223847] LustreError: 407305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.252207] LustreError: 407305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12150.451766] LustreError: 407285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275715 with bad export cookie 5269387966120115988 [12150.463761] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12150.729246] Lustre: server umount lustre-MDT0001 complete [12162.632514] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12167.278086] Key type lgssc unregistered [12167.612239] LNet: 411043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12167.626469] LNetError: 411043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12167.651389] LNet: Removed LNI 192.168.202.151@tcp [12169.265090] Key type .llcrypt unregistered [12169.268919] Key type ._llcrypt unregistered [12203.752308] Key type ._llcrypt registered [12203.754350] Key type .llcrypt registered [12203.845989] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12204.115334] Key type .llcrypt unregistered [12204.117082] Key type ._llcrypt unregistered [12230.674290] Key type ._llcrypt registered [12230.675604] Key type .llcrypt registered [12230.877492] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12234.037864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12234.087230] alg: No test for adler32 (adler32-zlib) [12235.224092] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [12235.537375] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12237.279182] Key type lgssc registered [12238.871050] Lustre: Echo OBD driver; http://www.lustre.org/ [12250.163765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12250.204256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12251.979905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12257.771593] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12262.378143] LustreError: 413256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12262.408553] LustreError: 413256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12267.399989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12267.494987] LustreError: 413255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12267.816083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12273.131674] LustreError: 413256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12273.143732] LustreError: 413256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12275.348527] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12277.220266] LustreError: 413255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.242514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12277.246776] LustreError: 413255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12282.338265] LustreError: 413252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.377581] LustreError: 413252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12284.654607] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12292.244638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.678528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12298.123298] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12313.057796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12313.061107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12313.082927] LustreError: Skipped 1 previous similar message [12313.084881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12313.139458] Lustre: Skipped 1 previous similar message [12314.885489] Lustre: server umount lustre-OST0000 complete [12319.929072] Lustre: server umount lustre-MDT0000 complete [12323.296836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.312791] LustreError: 415207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12323.326200] LustreError: 415207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12324.579094] LustreError: 413235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275890 with bad export cookie 6366465985931100069 [12324.601450] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12324.611820] LustreError: 413235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12325.173664] Lustre: server umount lustre-MDT0001 complete [12342.769930] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12345.340311] Key type lgssc unregistered [12345.815653] LNet: 416415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12345.825853] LNetError: 416415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12345.839524] LNet: Removed LNI 192.168.202.151@tcp [12346.506427] Key type .llcrypt unregistered [12346.508540] Key type ._llcrypt unregistered [12377.543906] Key type ._llcrypt registered [12377.546341] Key type .llcrypt registered [12377.656951] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12379.045574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12379.168929] alg: No test for adler32 (adler32-zlib) [12380.267509] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [12380.547412] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12382.321149] Key type lgssc registered [12383.682539] Lustre: Echo OBD driver; http://www.lustre.org/ [12392.802563] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12392.832353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12408.415585] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12413.535412] LustreError: 417801:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.151@tcp: failed processing log, type 4: rc = -110 [12440.223202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12446.933269] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12455.724729] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12465.064562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12465.807403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12465.924053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12471.265049] LustreError: 417827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12471.304392] LustreError: 417827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12471.931750] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12476.384828] LustreError: 417826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12476.400662] LustreError: 417826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12481.513050] LustreError: 417827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12481.524916] LustreError: 417827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12482.333131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.631889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12488.045434] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12498.433356] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12507.171862] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12521.962089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12521.976688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12522.006437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12523.493150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12523.496137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12523.555967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12525.028564] Lustre: server umount lustre-OST0000 complete [12533.733764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12533.737045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12533.786806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12533.831753] Lustre: Skipped 2 previous similar messages [12537.865896] Lustre: server umount lustre-MDT0000 complete [12538.871764] LustreError: 418843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12538.908272] LustreError: 418843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12543.041117] LustreError: 417807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276108 with bad export cookie 2901534537630997532 [12543.044059] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12543.050834] LustreError: 417807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12543.404913] Lustre: server umount lustre-MDT0001 complete [12561.035357] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12565.513166] Key type lgssc unregistered [12565.855938] LNet: 421376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12565.862307] LNetError: 421376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12565.900217] LNet: Removed LNI 192.168.202.151@tcp [12566.970768] Key type .llcrypt unregistered [12566.973425] Key type ._llcrypt unregistered [12600.282285] Key type ._llcrypt registered [12600.285879] Key type .llcrypt registered [12600.470756] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12602.320798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12602.348880] alg: No test for adler32 (adler32-zlib) [12603.531992] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [12603.964415] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12605.784054] Key type lgssc registered [12607.286739] Lustre: Echo OBD driver; http://www.lustre.org/ [12617.303511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12617.320764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12618.984915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12624.803096] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12629.484790] LustreError: 422794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12629.509387] LustreError: 422794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12634.611816] LustreError: 422795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12634.916520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12635.419648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12640.051803] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12649.781867] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12656.893777] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12666.740642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12667.008787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12675.083629] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12685.036030] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12689.542614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12695.012051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12695.015378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12695.026054] Lustre: Skipped 2 previous similar messages [12695.034607] Lustre: Skipped 1 previous similar message [12697.363103] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [12700.137063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12700.147393] Lustre: Skipped 2 previous similar messages [12701.281590] Lustre: server umount lustre-MDT0000 complete [12702.442041] LustreError: 422791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12704.985221] LustreError: 422776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.51@tcp arrived at 1789276270 with bad export cookie 460441086936362071 [12705.249684] LustreError: 422791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12705.258734] LustreError: 422791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12707.639338] LustreError: 422774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276273 with bad export cookie 460441086936361406 [12707.654793] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12707.660618] LustreError: 422774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12710.372858] LustreError: 422794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12710.383962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12710.399470] LustreError: 422794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12710.416567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12713.902150] Lustre: server umount lustre-OST0000 complete [12722.657562] LustreError: 422795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12722.694966] LustreError: 422795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12723.608254] Lustre: server umount lustre-MDT0001 complete [12739.324734] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12743.088993] Key type lgssc unregistered [12743.555670] LNet: 426483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12743.564850] LNetError: 426483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12743.584433] LNet: Removed LNI 192.168.202.151@tcp [12744.757208] Key type .llcrypt unregistered [12744.760866] Key type ._llcrypt unregistered [12776.169857] Key type ._llcrypt registered [12776.174107] Key type .llcrypt registered [12776.339786] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12778.188683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12778.283196] alg: No test for adler32 (adler32-zlib) [12779.571961] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [12780.015974] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12781.704074] Key type lgssc registered [12783.096032] Lustre: Echo OBD driver; http://www.lustre.org/ [12793.424919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12793.461619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12795.054655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12800.629280] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12804.584963] LustreError: 427900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12804.604043] LustreError: 427900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12809.695903] LustreError: 427899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12810.889614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12811.452330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12816.696066] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12825.568144] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12832.558179] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12840.818878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12841.290136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12848.749885] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12857.196519] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12864.909788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12873.184684] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12873.189908] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12873.227167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12874.541797] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [12875.240866] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12879.108766] Lustre: server umount lustre-MDT0001 complete [12879.608674] LustreError: 430402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12885.479958] LustreError: 428628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12885.509464] LustreError: 428628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12890.592314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12890.605223] LustreError: 427900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12890.607196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12890.612242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12890.665656] Lustre: Skipped 2 previous similar messages [12893.377537] Lustre: server umount lustre-OST0000 complete [12899.032238] Lustre: server umount lustre-MDT0000 complete [12916.166291] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12919.238735] Key type lgssc unregistered [12919.645766] LNet: 431608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12919.655949] LNetError: 431608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12920.690874] LNet: Removed LNI 192.168.202.151@tcp [12921.733408] Key type .llcrypt unregistered [12921.735310] Key type ._llcrypt unregistered [12949.327189] Key type ._llcrypt registered [12949.332945] Key type .llcrypt registered [12949.469729] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [12966.129625] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12967.686972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12967.737061] alg: No test for adler32 (adler32-zlib) [12968.748233] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [12968.979913] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12970.696091] Key type lgssc registered [12971.931490] Lustre: Echo OBD driver; http://www.lustre.org/ [13015.636774] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13035.192755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13035.238405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13036.549438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13036.591678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13036.702210] Lustre: lustre-MDT0000: new disk, initializing [13036.780454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13036.794732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13041.677450] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13058.862658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13059.235741] Lustre: lustre-OST0000: new disk, initializing [13059.239554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13059.244916] Lustre: Skipped 1 previous similar message [13059.250306] Lustre: 436654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13059.312663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13066.348636] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13068.902517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13068.908949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13068.935477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13076.865764] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13094.226319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13094.458679] Lustre: 435636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13094.552951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13094.698992] Lustre: lustre-MDT0001: new disk, initializing [13094.843313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13094.897308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13094.908528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13100.814659] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13112.907760] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13113.346030] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13119.604361] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13119.945944] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13127.652202] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13127.659021] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13127.667742] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13127.670869] Lustre: Skipped 1 previous similar message [13130.502362] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [13130.519207] Lustre: Skipped 1 previous similar message [13131.642594] Lustre: server umount lustre-MDT0001 complete [13137.900522] LustreError: 435643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13137.934537] LustreError: 435643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13143.012735] LustreError: 437112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13143.022207] LustreError: 437112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13148.130824] LustreError: 437112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13148.145470] LustreError: 437112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13150.691519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13150.698405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13150.726801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13153.252056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13153.252855] LustreError: 439285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13156.476330] Lustre: server umount lustre-OST0000 complete [13158.368313] LustreError: 435648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13166.508289] Lustre: server umount lustre-MDT0000 complete [13182.397621] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13185.151149] Key type lgssc unregistered [13185.400587] LNet: 440404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13185.414295] LNetError: 440404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13185.432463] LNet: Removed LNI 192.168.202.151@tcp [13186.563114] Key type .llcrypt unregistered [13186.568190] Key type ._llcrypt unregistered [13236.970309] Key type ._llcrypt registered [13236.975325] Key type .llcrypt registered [13237.119161] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13237.767386] Key type .llcrypt unregistered [13237.770256] Key type ._llcrypt unregistered [13262.032848] Key type ._llcrypt registered [13262.037247] Key type .llcrypt registered [13262.229141] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13264.108459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13264.157850] alg: No test for adler32 (adler32-zlib) [13265.310065] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [13265.641304] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [13267.463203] Key type lgssc registered [13269.273354] Lustre: Echo OBD driver; http://www.lustre.org/ [13299.381175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13299.410102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13300.667822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13300.689865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13300.747269] Lustre: lustre-MDT0000: new disk, initializing [13300.830250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13300.851528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13305.814622] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13317.773858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13317.916582] Lustre: 443774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13317.967167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13317.970937] Lustre: Skipped 1 previous similar message [13318.030624] Lustre: lustre-MDT0001: new disk, initializing [13318.110416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13318.159388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13318.172448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13323.464671] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13334.886804] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13343.762243] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13353.228982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13353.861123] Lustre: lustre-OST0000: new disk, initializing [13353.870610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13353.881723] Lustre: 445744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13354.022429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13360.737563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13360.754983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13360.944365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13361.981606] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13373.811226] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13381.602155] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13381.615338] Lustre: Skipped 2 previous similar messages [13381.619991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13381.625582] Lustre: Skipped 1 previous similar message [13386.733493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13386.745263] Lustre: Skipped 2 previous similar messages [13388.075503] Lustre: server umount lustre-MDT0000 complete [13392.091940] LustreError: 443767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276957 with bad export cookie 2507101725076409821 [13392.096252] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13392.098537] LustreError: 443767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13392.580711] Lustre: server umount lustre-MDT0001 complete [13405.868782] Lustre: server umount lustre-OST0000 complete [13418.676501] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:23:02 (1789276982) [13427.657672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13428.197379] LustreError: 447771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13428.212481] LustreError: 447771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13428.304457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13433.007365] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13433.325675] LustreError: 447772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13438.440055] LustreError: 447771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13442.006295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13442.416905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13447.983213] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13457.854247] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13464.384497] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13484.078123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13484.309571] Lustre: Found index 0 for lustre-OST0000, updating log [13484.556339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13491.591722] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13498.934494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13500.283975] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13507.041875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13507.049840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13507.068367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13512.428415] Lustre: server umount lustre-OST0000 complete [13529.569818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13529.570532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13529.595153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13529.620832] Lustre: Skipped 2 previous similar messages [13529.638451] Lustre: Skipped 4 previous similar messages [13533.732157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13533.745646] Lustre: Skipped 1 previous similar message [13542.111290] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp [13543.391143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13543.706806] Lustre: server umount lustre-MDT0000 complete [13544.941577] LustreError: 449592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13544.971250] LustreError: 449592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13548.078269] LustreError: 447752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789277113 with bad export cookie 2507101725076411214 [13548.081943] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13548.099497] LustreError: 447752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13548.572128] Lustre: server umount lustre-MDT0001 complete [13559.872914] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:25:22 (1789277122) [14189.747147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14190.217401] LustreError: 454187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14190.358959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14195.692146] LustreError: 454188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14195.877329] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14199.780884] LustreError: 454187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14205.660651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14205.960655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14211.018096] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14219.575752] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14225.551076] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14246.571778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14246.623586] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp [14246.845410] Lustre: Found index 0 for lustre-OST0000, updating log [14247.084608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14254.190047] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14262.834108] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14266.928853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14266.951034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14266.972204] Lustre: Skipped 4 previous similar messages [14280.159150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14280.580867] Lustre: server umount lustre-OST0000 complete [14286.755358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14286.777059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14286.785407] Lustre: Skipped 1 previous similar message [14286.810742] Lustre: Skipped 1 previous similar message [14288.867736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14288.887055] Lustre: Skipped 3 previous similar messages [14292.268327] Lustre: server umount lustre-MDT0000 complete [14293.026121] LustreError: 454188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14293.048677] LustreError: 454188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14297.124261] LustreError: 454182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14297.152457] LustreError: 454182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [14297.496380] LustreError: 454168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789277862 with bad export cookie 2507101725076412299 [14297.507463] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14297.518640] LustreError: 454168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14298.035961] Lustre: server umount lustre-MDT0001 complete [14307.040501] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp [14336.923395] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [14348.974597] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14391.259673] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:39:14 (1789277954) [14415.743790] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [14419.616591] Key type lgssc unregistered [14419.995436] LNet: 460811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14420.013158] LNetError: 460811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14509.161328] LNet: Removed LNI 192.168.202.151@tcp [14509.955883] Key type .llcrypt unregistered [14509.957815] Key type ._llcrypt unregistered [14526.364582] Key type ._llcrypt registered [14526.370431] Key type .llcrypt registered [14526.486921] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14527.796859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14527.886277] alg: No test for adler32 (adler32-zlib) [14528.969831] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [14529.202047] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [14530.879366] Key type lgssc registered [14531.930442] Lustre: Echo OBD driver; http://www.lustre.org/ [14536.863585] LNet: Added LNI 192.168.202.151@tcp100 [8/256/0/180] [14537.633993] LNet: Added LNI 192.168.202.151@tcp101 [8/256/0/180] [14538.622277] LNet: Added LNI 192.168.202.151@tcp102 [8/256/0/180] [14539.539505] LNet: Added LNI 192.168.202.151@tcp103 [8/256/0/180] [14540.617665] LNet: Added LNI 192.168.202.151@tcp104 [8/256/0/180] [14541.641547] LNet: Added LNI 192.168.202.151@tcp105 [8/256/0/180] [14542.609921] LNet: Added LNI 192.168.202.151@tcp106 [8/256/0/180] [14543.692023] LNet: Added LNI 192.168.202.151@tcp107 [8/256/0/180] [14544.621564] LNet: Added LNI 192.168.202.151@tcp108 [8/256/0/180] [14545.555496] LNet: Added LNI 192.168.202.151@tcp109 [8/256/0/180] [14546.560905] LNet: Added LNI 192.168.202.151@tcp110 [8/256/0/180] [14547.600571] LNet: Added LNI 192.168.202.151@tcp111 [8/256/0/180] [14548.412098] LNet: Added LNI 192.168.202.151@tcp112 [8/256/0/180] [14549.392427] LNet: Added LNI 192.168.202.151@tcp113 [8/256/0/180] [14550.277346] LNet: Added LNI 192.168.202.151@tcp114 [8/256/0/180] [14551.239380] LNet: Added LNI 192.168.202.151@tcp115 [8/256/0/180] [14552.259688] LNet: Added LNI 192.168.202.151@tcp116 [8/256/0/180] [14553.176660] LNet: Added LNI 192.168.202.151@tcp117 [8/256/0/180] [14554.209668] LNet: Added LNI 192.168.202.151@tcp118 [8/256/0/180] [14555.230569] LNet: Added LNI 192.168.202.151@tcp119 [8/256/0/180] [14556.044191] LNet: Added LNI 192.168.202.151@tcp120 [8/256/0/180] [14556.973597] LNet: Added LNI 192.168.202.151@tcp121 [8/256/0/180] [14558.353118] LNet: Added LNI 192.168.202.151@tcp122 [8/256/0/180] [14559.438375] LNet: Added LNI 192.168.202.151@tcp123 [8/256/0/180] [14560.494670] LNet: Added LNI 192.168.202.151@tcp124 [8/256/0/180] [14561.709655] LNet: Added LNI 192.168.202.151@tcp125 [8/256/0/180] [14562.839959] LNet: Added LNI 192.168.202.151@tcp126 [8/256/0/180] [14563.927845] LNet: Added LNI 192.168.202.151@tcp127 [8/256/0/180] [14565.036706] LNet: Added LNI 192.168.202.151@tcp128 [8/256/0/180] [14565.987095] LNet: Added LNI 192.168.202.151@tcp129 [8/256/0/180] [14566.798723] LNet: Added LNI 192.168.202.151@tcp130 [8/256/0/180] [14567.689915] LNet: Added LNI 192.168.202.151@tcp131 [8/256/0/180] [14568.610981] LNet: Added LNI 192.168.202.151@tcp132 [8/256/0/180] [14569.921828] LNet: Added LNI 192.168.202.151@tcp133 [8/256/0/180] [14571.224926] LNet: Added LNI 192.168.202.151@tcp134 [8/256/0/180] [14572.347455] LNet: Added LNI 192.168.202.151@tcp135 [8/256/0/180] [14581.103347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14581.142571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14582.246416] Lustre: 463836:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14582.325336] LNetError: 461291:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14582.339797] LNetError: Protocol error connecting to 192.168.202.151@tcp115 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14582.355479] LNetError: Skipped 5 previous similar messages [14582.395410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14582.410809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14582.458268] Lustre: lustre-MDT0000: new disk, initializing [14582.526243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14582.535474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14582.560094] LNet: 3 local NIs in recovery (showing 3): 192.168.202.151@tcp125, 192.168.202.151@tcp126, 192.168.202.151@tcp129 [14583.392571] LNetError: 461291:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14583.422116] LNetError: 461291:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14583.444767] LNetError: Protocol error connecting to 192.168.202.151@tcp106 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14585.709460] LNetError: 463859:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14585.733207] LNetError: 463859:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14585.747090] LNetError: Protocol error connecting to 192.168.202.151@tcp126 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14585.757088] LNetError: Skipped 3 previous similar messages [14589.083149] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14589.794278] LNetError: 463861:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14589.811343] LNetError: Protocol error connecting to 192.168.202.151@tcp126 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14597.025198] LNetError: 461291:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14597.053187] LNetError: Protocol error connecting to 192.168.202.151@tcp126 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14601.936364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14602.148780] Lustre: 463853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14602.230816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14602.235486] Lustre: Skipped 1 previous similar message [14602.325098] Lustre: lustre-MDT0001: new disk, initializing [14602.404023] Lustre: 464654:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14602.419481] Lustre: 464654:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14602.607677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14602.664211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14602.678531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14608.939733] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14609.473898] Lustre: 464936:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14609.481241] Lustre: 464936:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14613.408865] LNetError: 461293:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14613.420989] LNetError: Protocol error connecting to 192.168.202.151@tcp126 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14620.735802] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14627.242049] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14630.566809] LNetError: 463861:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14630.577092] LNetError: 463861:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14630.587846] LNetError: Protocol error connecting to 192.168.202.151@tcp124 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14630.601611] LNetError: Skipped 7 previous similar messages [14634.620435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14635.207301] Lustre: lustre-OST0000: new disk, initializing [14635.213471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14635.225843] Lustre: 465828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14635.370990] Lustre: 465833:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14635.389726] Lustre: 465833:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14635.470863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14643.103242] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp126 [14643.260708] Lustre: 466125:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14643.286389] Lustre: 466125:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14643.381658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14643.389463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14643.600532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14644.561547] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14657.147995] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14669.300213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14669.312757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14669.313042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14669.320693] Lustre: Skipped 1 previous similar message [14669.336882] Lustre: Skipped 1 previous similar message [14670.498259] LNetError: 463866:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14670.509410] LNetError: 463866:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [14670.520391] LNetError: Protocol error connecting to 192.168.202.151@tcp106 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14670.533388] LNetError: Skipped 16 previous similar messages [14672.956220] Lustre: server umount lustre-OST0000 complete [14679.528898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14679.534373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14679.538042] LustreError: Skipped 1 previous similar message [14679.538871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14679.576969] Lustre: Skipped 1 previous similar message [14684.641619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14684.650309] Lustre: Skipped 2 previous similar messages [14688.737452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14688.745041] Lustre: Skipped 1 previous similar message [14691.295293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14691.645768] Lustre: server umount lustre-MDT0000 complete [14695.371442] LustreError: 463844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789278260 with bad export cookie 14000548924822745713 [14695.379949] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14695.381930] LustreError: 463844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14695.775513] Lustre: server umount lustre-MDT0001 complete [14708.949753] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [14711.673862] Key type lgssc unregistered [14712.080586] LNet: 467744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14712.089875] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14712.099273] LNet: Removed LNI 192.168.202.151@tcp [14713.119233] LNet: Removed LNI 192.168.202.151@tcp100 [14713.132459] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14713.144772] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14713.164662] LNet: Removed LNI 192.168.202.151@tcp101 [14714.207597] LNet: Removed LNI 192.168.202.151@tcp102 [14714.213478] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14714.230520] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14714.244494] LNet: Removed LNI 192.168.202.151@tcp103 [14715.295496] LNet: Removed LNI 192.168.202.151@tcp104 [14716.320776] LNet: Removed LNI 192.168.202.151@tcp105 [14716.325539] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14716.337287] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14716.354845] LNet: Removed LNI 192.168.202.151@tcp106 [14717.407859] LNet: Removed LNI 192.168.202.151@tcp107 [14718.431509] LNet: Removed LNI 192.168.202.151@tcp108 [14719.458264] LNet: Removed LNI 192.168.202.151@tcp109 [14720.479204] LNet: Removed LNI 192.168.202.151@tcp110 [14720.482497] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14720.501934] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14720.515945] LNet: Removed LNI 192.168.202.151@tcp111 [14721.567421] LNet: Removed LNI 192.168.202.151@tcp112 [14722.591336] LNet: Removed LNI 192.168.202.151@tcp113 [14723.615259] LNet: Removed LNI 192.168.202.151@tcp114 [14724.639213] LNet: Removed LNI 192.168.202.151@tcp115 [14725.663287] LNet: Removed LNI 192.168.202.151@tcp116 [14726.687138] LNet: Removed LNI 192.168.202.151@tcp117 [14727.711393] LNet: Removed LNI 192.168.202.151@tcp118 [14728.735358] LNet: Removed LNI 192.168.202.151@tcp119 [14728.750448] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14728.760980] LNetError: 467744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14728.772748] LNet: Removed LNI 192.168.202.151@tcp120 [14729.823732] LNet: Removed LNI 192.168.202.151@tcp121 [14730.853452] LNet: Removed LNI 192.168.202.151@tcp122 [14731.871755] LNet: Removed LNI 192.168.202.151@tcp123 [14732.895231] LNet: Removed LNI 192.168.202.151@tcp124 [14733.919242] LNet: Removed LNI 192.168.202.151@tcp125 [14733.923609] LNet: Removed LNI 192.168.202.151@tcp126 [14733.931994] LNet: Removed LNI 192.168.202.151@tcp127 [14734.943488] LNet: Removed LNI 192.168.202.151@tcp128 [14735.967222] LNet: Removed LNI 192.168.202.151@tcp129 [14735.973071] LNet: Removed LNI 192.168.202.151@tcp130 [14735.982360] LNet: Removed LNI 192.168.202.151@tcp131 [14735.987859] LNet: Removed LNI 192.168.202.151@tcp132 [14735.995562] LNet: Removed LNI 192.168.202.151@tcp133 [14735.997207] LNet: Removed LNI 192.168.202.151@tcp134 [14736.022675] LNet: Removed LNI 192.168.202.151@tcp135 [14736.756339] Key type .llcrypt unregistered [14736.758613] Key type ._llcrypt unregistered [14763.292138] Key type ._llcrypt registered [14763.294360] Key type .llcrypt registered [14763.381154] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [14780.666790] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14782.366647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14782.401962] alg: No test for adler32 (adler32-zlib) [14783.512996] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [14783.741664] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [14785.458392] Key type lgssc registered [14786.782811] Lustre: Echo OBD driver; http://www.lustre.org/ [14830.981255] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:46:34 (1789278394) [14853.228324] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [14856.853554] Key type lgssc unregistered [14857.209299] LNet: 471841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14857.219995] LNetError: 471841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14857.244486] LNet: Removed LNI 192.168.202.151@tcp [14858.414655] Key type .llcrypt unregistered [14858.416501] Key type ._llcrypt unregistered [14881.206245] Key type ._llcrypt registered [14881.207877] Key type .llcrypt registered [14881.282704] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14882.039832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14882.220885] alg: No test for adler32 (adler32-zlib) [14883.470097] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [14883.888157] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [14885.647234] Key type lgssc registered [14887.153735] Lustre: Echo OBD driver; http://www.lustre.org/ [14897.951717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14897.970039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14899.378469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14899.479056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14899.706459] Lustre: lustre-MDT0000: new disk, initializing [14899.869843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14899.911049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14905.524230] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14917.308385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14917.468567] Lustre: 473049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14917.569214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14917.574089] Lustre: Skipped 1 previous similar message [14917.682508] Lustre: lustre-MDT0001: new disk, initializing [14917.784178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14917.827699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14917.839120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14924.210231] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14934.645912] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14939.686673] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14946.608112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14946.965682] Lustre: lustre-OST0000: new disk, initializing [14946.969127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14946.980500] Lustre: 475018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14947.137618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14952.847756] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14955.589365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14955.597325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14955.752152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14962.761695] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14966.966319] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing lnet_if_list [14972.715837] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14972.776468] LNet: Added LNI 192.168.202.151@tcp10 [8/256/0/180] [14979.431355] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14979.513389] LNet: Added LNI 192.168.202.151@tcp11 [8/256/0/180] [14986.589465] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14986.652439] LNet: Added LNI 192.168.202.151@tcp12 [8/256/0/180] [14995.239737] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14995.302517] LNet: Added LNI 192.168.202.151@tcp13 [8/256/0/180] [15002.205983] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15002.253201] LNet: Added LNI 192.168.202.151@tcp14 [8/256/0/180] [15010.127808] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15010.192305] LNet: Added LNI 192.168.202.151@tcp15 [8/256/0/180] [15015.515180] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15015.558903] LNet: Added LNI 192.168.202.151@tcp16 [8/256/0/180] [15021.302222] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15021.326479] LNet: Added LNI 192.168.202.151@tcp17 [8/256/0/180] [15026.823844] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15026.864286] LNet: Added LNI 192.168.202.151@tcp18 [8/256/0/180] [15031.993921] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15032.017760] LNet: Added LNI 192.168.202.151@tcp19 [8/256/0/180] [15037.135569] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15037.156543] LNet: Added LNI 192.168.202.151@tcp20 [8/256/0/180] [15042.656463] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15042.682745] LNet: Added LNI 192.168.202.151@tcp21 [8/256/0/180] [15048.973687] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15049.005258] LNet: Added LNI 192.168.202.151@tcp22 [8/256/0/180] [15055.222356] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15055.262799] LNet: Added LNI 192.168.202.151@tcp23 [8/256/0/180] [15061.603814] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15061.627646] LNet: Added LNI 192.168.202.151@tcp24 [8/256/0/180] [15067.262207] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15067.290897] LNet: Added LNI 192.168.202.151@tcp25 [8/256/0/180] [15072.558501] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15072.581159] LNet: Added LNI 192.168.202.151@tcp26 [8/256/0/180] [15078.692167] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15078.722974] LNet: Added LNI 192.168.202.151@tcp27 [8/256/0/180] [15085.381305] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15085.422300] LNet: Added LNI 192.168.202.151@tcp28 [8/256/0/180] [15092.623907] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15092.679765] LNet: Added LNI 192.168.202.151@tcp29 [8/256/0/180] [15100.288048] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15100.333944] LNet: Added LNI 192.168.202.151@tcp30 [8/256/0/180] [15107.058116] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15107.078161] LNet: Added LNI 192.168.202.151@tcp31 [8/256/0/180] [15113.929794] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15113.957818] LNet: Added LNI 192.168.202.151@tcp32 [8/256/0/180] [15121.265150] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15121.297739] LNet: Added LNI 192.168.202.151@tcp33 [8/256/0/180] [15127.968838] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15127.992868] LNet: Added LNI 192.168.202.151@tcp34 [8/256/0/180] [15134.333831] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15134.363325] LNet: Added LNI 192.168.202.151@tcp35 [8/256/0/180] [15142.527281] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15142.558343] LNet: Added LNI 192.168.202.151@tcp36 [8/256/0/180] [15150.261271] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15150.296847] LNet: Added LNI 192.168.202.151@tcp37 [8/256/0/180] [15157.495314] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15157.559363] LNet: Added LNI 192.168.202.151@tcp38 [8/256/0/180] [15165.574318] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15165.612273] LNet: Added LNI 192.168.202.151@tcp39 [8/256/0/180] [15174.217529] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15174.247603] LNet: Added LNI 192.168.202.151@tcp40 [8/256/0/180] [15181.753844] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15181.780615] LNet: Added LNI 192.168.202.151@tcp41 [8/256/0/180] [15191.214946] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15191.276235] LNet: Added LNI 192.168.202.151@tcp42 [8/256/0/180] [15199.375449] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15199.427478] LNet: Added LNI 192.168.202.151@tcp43 [8/256/0/180] [15207.809668] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15207.833197] LNet: Added LNI 192.168.202.151@tcp44 [8/256/0/180] [15216.398469] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15216.424814] LNet: Added LNI 192.168.202.151@tcp45 [8/256/0/180] [15223.748412] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15223.810977] LNet: Added LNI 192.168.202.151@tcp46 [8/256/0/180] [15232.034234] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15232.059427] LNet: Added LNI 192.168.202.151@tcp47 [8/256/0/180] [15239.613790] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15239.636831] LNet: Added LNI 192.168.202.151@tcp48 [8/256/0/180] [15247.902709] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15247.951060] LNet: Added LNI 192.168.202.151@tcp49 [8/256/0/180] [15257.007492] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15257.031832] LNet: Added LNI 192.168.202.151@tcp50 [8/256/0/180] [15276.515750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15276.530127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15276.557530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15278.563905] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15278.568624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15278.592518] Lustre: Skipped 2 previous similar messages [15282.805237] Lustre: server umount lustre-MDT0000 complete [15286.598384] LustreError: 473043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789278852 with bad export cookie 8399296817222517363 [15286.603071] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15286.614558] LustreError: 473043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15287.194752] Lustre: server umount lustre-MDT0001 complete [15305.188421] Lustre: 472324:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789278854/real 1789278854] req@000000002de652fb x1876194439119744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789278870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15305.216471] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15305.227237] Lustre: Skipped 1 previous similar message [15306.222806] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15308.258338] Lustre: 472323:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789278857/real 1789278857] req@000000001b95b6d3 x1876194439119872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789278873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15310.306736] Lustre: 472321:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789278859/real 1789278859] req@00000000bbba669a x1876194439120000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789278875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15313.375193] Lustre: 472322:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789278862/real 1789278862] req@0000000093eb8298 x1876194439120256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789278878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15315.023828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15318.495349] Lustre: 472323:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789278867/real 1789278867] req@00000000c506a3f4 x1876194439120384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789278883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15331.871613] LustreError: 472320:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000048607c55 x1876194439121152/t0(0) o250->MGC192.168.202.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15332.228201] LustreError: 475012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15332.260067] LustreError: 475012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15332.287503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15334.305637] LustreError: 475013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15334.306578] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15337.564331] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15339.490341] LustreError: 475012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15344.629520] LustreError: 475013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15348.102915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15348.401652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15350.448493] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15352.903782] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15361.746188] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15367.804756] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15389.666445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15389.676663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15389.701198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15391.716943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15391.721233] Lustre: Skipped 1 previous similar message [15395.218739] Lustre: server umount lustre-OST0000 complete [15399.392242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15399.396824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15399.414124] Lustre: Skipped 1 previous similar message [15399.423901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15405.663574] Lustre: server umount lustre-MDT0000 complete [15407.073855] LustreError: 499262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15407.097234] LustreError: 499262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15409.529454] LustreError: 473042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789278975 with bad export cookie 8399296817222520128 [15409.532578] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15409.570738] LustreError: 473042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15409.883202] Lustre: server umount lustre-MDT0001 complete [15420.457630] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15423.643451] Key type lgssc unregistered [15424.147986] LNet: 502063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15424.168480] LNetError: 502063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15424.177070] LNet: Removed LNI 192.168.202.151@tcp [15424.178857] LNet: Removed LNI 192.168.202.151@tcp10 [15424.184949] LNet: Removed LNI 192.168.202.151@tcp11 [15424.186802] LNet: Removed LNI 192.168.202.151@tcp12 [15424.203592] LNet: Removed LNI 192.168.202.151@tcp13 [15424.207143] LNet: Removed LNI 192.168.202.151@tcp14 [15424.213170] LNet: Removed LNI 192.168.202.151@tcp15 [15424.216365] LNet: Removed LNI 192.168.202.151@tcp16 [15424.219918] LNet: Removed LNI 192.168.202.151@tcp17 [15424.222349] LNet: Removed LNI 192.168.202.151@tcp18 [15424.225370] LNet: Removed LNI 192.168.202.151@tcp19 [15424.228159] LNet: Removed LNI 192.168.202.151@tcp20 [15424.231052] LNet: Removed LNI 192.168.202.151@tcp21 [15424.233538] LNet: Removed LNI 192.168.202.151@tcp22 [15424.236511] LNet: Removed LNI 192.168.202.151@tcp23 [15424.238913] LNet: Removed LNI 192.168.202.151@tcp24 [15424.241372] LNet: Removed LNI 192.168.202.151@tcp25 [15424.244656] LNet: Removed LNI 192.168.202.151@tcp26 [15424.247032] LNet: Removed LNI 192.168.202.151@tcp27 [15424.249552] LNet: Removed LNI 192.168.202.151@tcp28 [15424.253615] LNet: Removed LNI 192.168.202.151@tcp29 [15424.256151] LNet: Removed LNI 192.168.202.151@tcp30 [15424.259471] LNet: Removed LNI 192.168.202.151@tcp31 [15424.262358] LNet: Removed LNI 192.168.202.151@tcp32 [15424.264170] LNet: Removed LNI 192.168.202.151@tcp33 [15424.267236] LNet: Removed LNI 192.168.202.151@tcp34 [15424.271290] LNet: Removed LNI 192.168.202.151@tcp35 [15424.273882] LNet: Removed LNI 192.168.202.151@tcp36 [15424.276518] LNet: Removed LNI 192.168.202.151@tcp37 [15424.279279] LNet: Removed LNI 192.168.202.151@tcp38 [15424.283461] LNet: Removed LNI 192.168.202.151@tcp39 [15424.286323] LNet: Removed LNI 192.168.202.151@tcp40 [15424.288915] LNet: Removed LNI 192.168.202.151@tcp41 [15424.291410] LNet: Removed LNI 192.168.202.151@tcp42 [15424.294196] LNet: Removed LNI 192.168.202.151@tcp43 [15424.300897] LNet: Removed LNI 192.168.202.151@tcp44 [15424.303172] LNet: Removed LNI 192.168.202.151@tcp45 [15424.304975] LNet: Removed LNI 192.168.202.151@tcp46 [15424.307440] LNet: Removed LNI 192.168.202.151@tcp47 [15424.309908] LNet: Removed LNI 192.168.202.151@tcp48 [15424.311929] LNet: Removed LNI 192.168.202.151@tcp49 [15424.321951] LNet: Removed LNI 192.168.202.151@tcp50 [15425.093163] Key type .llcrypt unregistered [15425.094632] Key type ._llcrypt unregistered [15450.524732] Key type ._llcrypt registered [15450.527408] Key type .llcrypt registered [15450.640641] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [15469.788156] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15471.459325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15471.508617] alg: No test for adler32 (adler32-zlib) [15472.744885] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [15473.103337] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [15474.815273] Key type lgssc registered [15475.869406] Lustre: Echo OBD driver; http://www.lustre.org/ [15514.288804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15514.334048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15515.643601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15515.779347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15515.965192] Lustre: lustre-MDT0000: new disk, initializing [15516.172873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15516.208829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15521.534889] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15534.220135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15534.397849] Lustre: 505260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15534.498866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15534.503470] Lustre: Skipped 1 previous similar message [15534.674489] Lustre: lustre-MDT0001: new disk, initializing [15534.849309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15534.926705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15534.930778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15541.253286] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15552.844286] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15559.283627] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15568.648924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15568.834428] Lustre: *** cfs_fail_loc=90f, val=0*** [15568.986402] Lustre: lustre-OST0000: new disk, initializing [15568.988914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15568.997066] Lustre: 507281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15569.088809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15575.433318] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15578.209689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15578.221569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15578.346143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15585.900643] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15593.955099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15593.957906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15593.969709] LustreError: Skipped 1 previous similar message [15593.978092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15593.996871] Lustre: Skipped 1 previous similar message [15594.024280] Lustre: Skipped 1 previous similar message [15599.078318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15599.085181] Lustre: Skipped 1 previous similar message [15600.052936] Lustre: server umount lustre-OST0000 complete [15604.205120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15604.225716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15604.233950] Lustre: Skipped 1 previous similar message [15609.314958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15609.325121] Lustre: Skipped 1 previous similar message [15610.247531] Lustre: server umount lustre-MDT0000 complete [15613.758996] LustreError: 505252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789279179 with bad export cookie 10392972672612011027 [15613.777412] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15614.161084] Lustre: server umount lustre-MDT0001 complete [15625.260776] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15628.088347] Key type lgssc unregistered [15628.347980] LNet: 509185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15628.354438] LNetError: 509185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15628.379644] LNet: Removed LNI 192.168.202.151@tcp [15629.313639] Key type .llcrypt unregistered [15629.315497] Key type ._llcrypt unregistered [15649.329896] Key type ._llcrypt registered [15649.331474] Key type .llcrypt registered [15649.498411] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15651.443630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15651.475782] alg: No test for adler32 (adler32-zlib) [15652.461540] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [15652.650258] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [15654.327209] Key type lgssc registered [15655.383607] Lustre: Echo OBD driver; http://www.lustre.org/ [15663.012222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15663.039382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15664.461677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15669.100396] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15673.826287] LustreError: 510455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15673.843267] LustreError: 510455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15678.500522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15678.863935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15684.240361] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15693.704784] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15700.292801] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15708.175771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15708.522602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15715.375892] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15722.223827] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15735.531806] loop0: detected capacity change from 0 to 204800000 [15735.754587] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15735.766841] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15735.810489] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15742.051337] loop0: detected capacity change from 0 to 204800000 [15742.239865] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15742.248503] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15742.260234] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15747.465612] loop0: detected capacity change from 0 to 204800000 [15753.155285] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15753.298685] Lustre: fs15246-OST0000: new disk, initializing [15753.303736] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15753.312419] Lustre: 514061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15753.393727] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15760.403884] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15771.551817] loop1: detected capacity change from 0 to 204800000 [15778.240645] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15778.364857] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15778.488868] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15778.676197] Lustre: fs15246-MDT0000: new disk, initializing [15778.912286] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15778.939552] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15779.118299] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15780.199083] Lustre: 515184:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ffa056b3 x1876195245341952/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789279361 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15784.442120] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15789.365878] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15789.383626] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15789.415110] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15807.994910] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15808.019807] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15812.109880] Lustre: server umount fs15246-MDT0000 complete [15813.100181] LustreError: 510451:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15818.209866] LustreError: 510450:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15821.484211] Lustre: server umount fs15246-OST0000 complete [15836.643069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15836.649122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15836.666322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15837.664367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15837.674122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15837.701109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15843.810880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15843.819036] Lustre: Skipped 3 previous similar messages [15848.940235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15848.947059] Lustre: Skipped 1 previous similar message [15849.951886] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15850.179195] Lustre: server umount lustre-OST0000 complete [15854.049736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15854.066857] Lustre: Skipped 1 previous similar message [15859.168355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15859.172038] Lustre: Skipped 3 previous similar messages [15868.388393] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15868.600090] Lustre: server umount lustre-MDT0000 complete [15869.417768] LustreError: 511744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15869.436884] LustreError: 511744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15872.221765] LustreError: 510436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789279437 with bad export cookie 8896876578121711270 [15872.224290] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15872.231252] LustreError: 510436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15872.575902] Lustre: server umount lustre-MDT0001 complete [15885.073705] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15887.697663] Key type lgssc unregistered [15888.030671] LNet: 518284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15888.039683] LNetError: 518284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15888.060627] LNet: Removed LNI 192.168.202.151@tcp [15888.909561] Key type .llcrypt unregistered [15888.911597] Key type ._llcrypt unregistered [15910.208033] Key type ._llcrypt registered [15910.211858] Key type .llcrypt registered [15910.334988] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15911.699736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15911.758356] alg: No test for adler32 (adler32-zlib) [15912.975401] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [15913.271171] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [15914.959184] Key type lgssc registered [15916.469974] Lustre: Echo OBD driver; http://www.lustre.org/ [15967.930112] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [15979.575814] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [16018.589042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16018.650050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16019.976589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16020.006990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16020.089501] Lustre: lustre-MDT0000: new disk, initializing [16020.146447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16020.154182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16024.732793] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16035.727582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16035.859560] Lustre: 522759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16035.910518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16035.913919] Lustre: Skipped 1 previous similar message [16035.991913] Lustre: lustre-MDT0001: new disk, initializing [16036.102266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16036.176296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16036.186174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16040.421469] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16050.118342] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16056.446576] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16063.777885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16064.082501] Lustre: lustre-OST0000: new disk, initializing [16064.085852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16064.092116] Lustre: 524726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16064.188679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16071.016229] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16071.757118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16071.774576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16071.832250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16081.064146] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16085.964169] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16086.162226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16090.823699] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16091.141167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16097.772355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16097.778866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16097.783576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16097.793693] Lustre: Skipped 1 previous similar message [16099.677123] Lustre: server umount lustre-OST0000 complete [16108.000398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16108.003423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16108.017932] LustreError: Skipped 1 previous similar message [16108.023062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16108.023070] Lustre: Skipped 1 previous similar message [16108.071502] Lustre: Skipped 1 previous similar message [16109.425626] Lustre: server umount lustre-MDT0000 complete [16113.292683] LustreError: 522749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789279678 with bad export cookie 17072667849305041318 [16113.314335] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16114.108325] Lustre: server umount lustre-MDT0001 complete [16137.091996] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [16146.377911] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [16175.890580] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16010 sec ========= 02:08:59 (1789279739) [16177.497839] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:09:01 (1789279741) === [16181.074345] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:09:04 (1789279744) === [16220.046465] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [16222.978940] Key type lgssc unregistered [16223.284650] LNet: 531366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16223.296467] LNetError: 531366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16223.311345] LNet: Removed LNI 192.168.202.151@tcp [16224.359345] Key type .llcrypt unregistered [16224.361653] Key type ._llcrypt unregistered