[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494430396 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003149] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008255] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011123] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022716] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027360] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] *** VALIDATE tmpfs *** [ 0.032210] *** VALIDATE proc *** [ 0.033269] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042340] debug: unmapping init [mem 0xffffffff96a59000-0xffffffff96a60fff] [ 0.044172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053305] rcu: Hierarchical SRCU implementation. [ 0.055400] smp: Bringing up secondary CPUs ... [ 0.056609] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.060265] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140018] node 0 deferred pages initialised in 74ms [ 0.143298] devtmpfs: initialized [ 0.144230] x86/mm: Memory block size: 128MB [ 0.146934] gcov: version magic: 0x41383552 [ 0.148156] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150266] pinctrl core: initialized pinctrl subsystem [ 0.151224] [ 0.151886] ************************************************************* [ 0.152020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153014] ** ** [ 0.154014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155017] ** ** [ 0.156015] ** This means that this kernel is built to expose internal ** [ 0.157017] ** IOMMU data structures, which may compromise security on ** [ 0.158013] ** your system. ** [ 0.159015] ** ** [ 0.160014] ** If you see this message and you are not debugging the ** [ 0.161016] ** kernel, report this immediately to your vendor! ** [ 0.162015] ** ** [ 0.163022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164014] ************************************************************* [ 0.165682] NET: Registered protocol family 16 [ 0.166454] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171123] cpuidle: using governor menu [ 0.172624] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173469] PCI: Using configuration type 1 for base access [ 0.174151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181040] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.182024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.185029] cryptd: max_cpu_qlen set to 1000 [ 0.187091] ACPI: Added _OSI(Module Device) [ 0.188026] ACPI: Added _OSI(Processor Device) [ 0.189000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.195162] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201587] ACPI: Interpreter enabled [ 0.203055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205017] ACPI: Using IOAPIC for interrupt routing [ 0.207145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.211479] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.220566] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.226018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233344] acpiphp: Slot [2] registered [ 0.234127] acpiphp: Slot [5] registered [ 0.236130] acpiphp: Slot [6] registered [ 0.237124] acpiphp: Slot [7] registered [ 0.238000] acpiphp: Slot [8] registered [ 0.239111] acpiphp: Slot [9] registered [ 0.240187] acpiphp: Slot [10] registered [ 0.242138] acpiphp: Slot [3] registered [ 0.243105] acpiphp: Slot [4] registered [ 0.245100] acpiphp: Slot [11] registered [ 0.246111] acpiphp: Slot [12] registered [ 0.248181] acpiphp: Slot [13] registered [ 0.249117] acpiphp: Slot [14] registered [ 0.250124] acpiphp: Slot [15] registered [ 0.252125] acpiphp: Slot [16] registered [ 0.254124] acpiphp: Slot [17] registered [ 0.255112] acpiphp: Slot [18] registered [ 0.256159] acpiphp: Slot [19] registered [ 0.258110] acpiphp: Slot [20] registered [ 0.259152] acpiphp: Slot [21] registered [ 0.260128] acpiphp: Slot [22] registered [ 0.262130] acpiphp: Slot [23] registered [ 0.263113] acpiphp: Slot [24] registered [ 0.265217] acpiphp: Slot [25] registered [ 0.268264] acpiphp: Slot [26] registered [ 0.270096] acpiphp: Slot [27] registered [ 0.271160] acpiphp: Slot [28] registered [ 0.272104] acpiphp: Slot [29] registered [ 0.274094] acpiphp: Slot [30] registered [ 0.275114] acpiphp: Slot [31] registered [ 0.276074] PCI host bridge to bus 0000:00 [ 0.278020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.280024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.282026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.285024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.287029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.290027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.292167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.294958] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.297256] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.307016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313078] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.315018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.317018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.320018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.322677] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.325832] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.328040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.330670] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.336013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.348018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.354017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.362217] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.370021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.396022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438125] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.454028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.510021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.531014] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.547035] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.560027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.591021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.607228] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.623018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.640019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.675020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.691118] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.714017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.739016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.768018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.786709] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.795024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.801026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.815041] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.824412] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.826381] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.828379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.831451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.833247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.839047] iommu: Default domain type: Passthrough [ 0.841670] SCSI subsystem initialized [ 0.842282] ACPI: bus type USB registered [ 0.845129] usbcore: registered new interface driver usbfs [ 0.846090] usbcore: registered new interface driver hub [ 0.848091] usbcore: registered new device driver usb [ 0.850166] pps_core: LinuxPPS API ver. 1 registered [ 0.851014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.854066] PTP clock support registered [ 0.857041] EDAC MC: Ver: 3.0.0 [ 0.859166] PCI: Using ACPI for IRQ routing [ 0.861463] NetLabel: Initializing [ 0.863018] NetLabel: domain hash size = 128 [ 0.864016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.865116] NetLabel: unlabeled traffic allowed by default [ 0.868060] vgaarb: loaded [ 0.869373] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.870022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.877014] clocksource: Switched to clocksource kvm-clock [ 0.993977] VFS: Disk quotas dquot_6.6.0 [ 0.995649] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.998091] *** VALIDATE ramfs *** [ 0.999401] *** VALIDATE hugetlbfs *** [ 1.001119] pnp: PnP ACPI init [ 1.003574] pnp: PnP ACPI: found 6 devices [ 1.021906] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.024343] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.025614] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.027026] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.028465] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.029859] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.031935] NET: Registered protocol family 2 [ 1.037230] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.043430] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.047531] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.053045] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.056476] TCP: Hash tables configured (established 65536 bind 65536) [ 1.059805] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.062713] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.065815] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.069298] NET: Registered protocol family 1 [ 1.071757] RPC: Registered named UNIX socket transport module. [ 1.074454] RPC: Registered udp transport module. [ 1.076240] RPC: Registered tcp transport module. [ 1.077832] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.080132] NET: Registered protocol family 44 [ 1.081628] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.083996] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.086203] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.088535] PCI: CLS 0 bytes, default 64 [ 1.090962] Unpacking initramfs... [ 2.486885] debug: unmapping init [mem 0xffff8a76fcc54000-0xffff8a76fffbffff] [ 2.490931] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.493595] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.496990] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.982209] Initialise system trusted keyrings [ 2.983499] Key type blacklist registered [ 2.985532] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.993917] zbud: loaded [ 2.996683] *** VALIDATE nfs *** [ 2.997486] *** VALIDATE nfs4 *** [ 2.998634] pstore: using deflate compression [ 3.003052] Platform Keyring initialized [ 3.115656] NET: Registered protocol family 38 [ 3.116956] Key type asymmetric registered [ 3.117825] Asymmetric key parser 'x509' registered [ 3.118944] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.120858] io scheduler mq-deadline registered [ 3.121779] io scheduler kyber registered [ 3.123656] io scheduler bfq registered [ 3.125471] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.127595] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.130377] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.133180] ACPI: Power Button [PWRF] [ 3.138310] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.144498] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.156680] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.163186] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.173328] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.200244] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.225019] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.228383] Non-volatile memory driver v1.3 [ 3.229277] Linux agpgart interface v0.103 [ 3.259056] virtio_blk virtio1: [vda] 145936 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.261567] vda: detected capacity change from 0 to 74719232 [ 3.276555] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.279243] vdb: detected capacity change from 0 to 1073741824 [ 3.302619] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.305153] vdc: detected capacity change from 0 to 2621440000 [ 3.319795] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.322262] vdd: detected capacity change from 0 to 2621440000 [ 3.337329] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.339657] vde: detected capacity change from 0 to 4294967296 [ 3.354822] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.357377] vdf: detected capacity change from 0 to 4294967296 [ 3.369030] libphy: Fixed MDIO Bus: probed [ 3.377974] usbcore: registered new interface driver usbserial_generic [ 3.379592] usbserial: USB Serial support registered for generic [ 3.380961] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.383665] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.384820] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.386426] mousedev: PS/2 mouse device common for all mice [ 3.388219] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.390443] rtc_cmos 00:05: RTC can wake from S4 [ 3.396514] rtc_cmos 00:05: registered as rtc0 [ 3.397512] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.399115] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.404790] intel_pstate: CPU model not supported [ 3.406877] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.410665] hid: raw HID events driver (C) Jiri Kosina [ 3.413346] usbcore: registered new interface driver usbhid [ 3.415473] usbhid: USB HID core driver [ 3.416861] drop_monitor: Initializing network drop monitor service [ 3.419725] Initializing XFRM netlink socket [ 3.421868] NET: Registered protocol family 10 [ 3.425361] Segment Routing with IPv6 [ 3.427026] NET: Registered protocol family 17 [ 3.428725] mpls_gso: MPLS GSO support [ 3.433626] RAS: Correctable Errors collector initialized. [ 3.435346] AVX version of gcm_enc/dec engaged. [ 3.436894] AES CTR mode by8 optimization enabled [ 3.523258] sched_clock: Marking stable (3523214814, 0)->(4461052150, -937837336) [ 3.526985] registered taskstats version 1 [ 3.529401] Loading compiled-in X.509 certificates [ 3.531404] zswap: loaded using pool lzo/zbud [ 3.557778] Key type big_key registered [ 3.569575] Key type encrypted registered [ 3.571230] ima: No TPM chip found, activating TPM-bypass! [ 3.572632] ima: Allocated hash algorithm: sha1 [ 3.573966] ima: No architecture policies found [ 3.575587] evm: Initialising EVM extended attributes: [ 3.577433] evm: security.selinux [ 3.578622] evm: security.ima [ 3.579660] evm: security.capability [ 3.580931] evm: HMAC attrs: 0x1 [ 3.583070] rtc_cmos 00:05: setting system clock to 2026-07-24 23:20:00 UTC (1784935200) [ 3.589436] debug: unmapping init [mem 0xffffffff97a03000-0xffffffff97bfffff] [ 3.592252] debug: unmapping init [mem 0xffffffff96782000-0xffffffff96a58fff] [ 3.604180] Write protecting the kernel read-only data: 28672k [ 3.606601] debug: unmapping init [mem 0xffffffff94e03000-0xffffffff94ffffff] [ 3.608215] debug: unmapping init [mem 0xffffffff95714000-0xffffffff957fffff] [ 3.650634] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.659645] systemd[1]: Detected virtualization kvm. [ 3.661385] systemd[1]: Detected architecture x86-64. [ 3.662756] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.686124] systemd[1]: No hostname configured. [ 3.687668] systemd[1]: Set hostname to . [ 3.689589] random: systemd: uninitialized urandom read (16 bytes read) [ 3.691372] systemd[1]: Initializing machine ID from random generator. [ 3.813706] random: systemd: uninitialized urandom read (16 bytes read) [ 3.815481] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.819507] random: systemd: uninitialized urandom read (16 bytes read) [ 3.821151] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.824432] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.369126] device-mapper: uevent: version 1.0.3 [ 4.370706] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.681937] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.006286] virtio_net virtio0 ens2: renamed from eth0 [ 5.059230] scsi host0: ata_piix [ 5.062870] scsi host1: ata_piix [ 5.064407] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.066797] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.724392] dracut-initqueue[576]: RTNETLINK answers: File exists [ 10.086298] random: crng init done [ 10.087678] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.363869] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.606206] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.824513] SELinux: Disabled at runtime. [ 11.875718] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.883852] systemd[1]: Detected virtualization kvm. [ 11.885740] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.302249] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.304543] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.309415] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.312401] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.314574] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.322822] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.330838] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.485107] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.759850] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.076657] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.117515] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.188957] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.211324] EDAC sbridge: Ver: 1.1.2 [ 14.543574] Key type dns_resolver registered [ 14.854285] NFS: Registering the id_resolver key type [ 14.856395] Key type id_resolver registered [ 14.858769] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg136-server login: [ 40.586518] libcfs: loading out-of-tree module taints kernel. [ 40.604051] Key type ._llcrypt registered [ 40.605094] Key type .llcrypt registered [ 40.641136] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 49.166897] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 49.756522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.763047] alg: No test for adler32 (adler32-zlib) [ 50.802298] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 51.149444] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 52.775182] Key type lgssc registered [ 53.473679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 57.282452] hrtimer: interrupt took 2393390 ns [ 73.550972] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 119.790644] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 135.382171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 135.402296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 136.666348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 136.706898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 136.781390] Lustre: lustre-MDT0000: new disk, initializing [ 136.871113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 136.898379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 142.061139] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.791906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.880840] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 155.917436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 155.921749] Lustre: Skipped 1 previous similar message [ 156.021106] Lustre: lustre-MDT0001: new disk, initializing [ 156.088361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.114298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.128298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 160.409295] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.011821] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 174.899812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 175.205923] Lustre: lustre-OST0000: new disk, initializing [ 175.211208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 175.218071] Lustre: 8421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 175.295163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.976494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.984771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 178.077318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 181.574344] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.486977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.668474] Lustre: lustre-OST0001: new disk, initializing [ 196.672561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 196.683505] Lustre: 9492:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.833298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.163365] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.393895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.409714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.470331] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.627784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.406758] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.235684] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing check_logdir /tmp/testlogs/ [ 238.998316] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing yml_node [ 243.639156] Lustre: DEBUG MARKER: Client: 2.17.54.207 [ 245.861968] Lustre: DEBUG MARKER: MDS: 2.17.54.207 [ 248.199835] Lustre: DEBUG MARKER: OSS: 2.17.54.207 [ 249.765314] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 24 19:24:04 EDT 2026 [ 266.149035] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 267.901269] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.225665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.240639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.251851] Lustre: Skipped 1 previous similar message [ 288.263469] Lustre: Skipped 3 previous similar messages [ 290.714569] Lustre: server umount lustre-MDT0000 complete [ 294.690430] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784935491 with bad export cookie 9377232003654874356 [ 294.692983] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.701174] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 298.465659] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.469708] LustreError: 12075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.479853] Lustre: Skipped 2 previous similar messages [ 298.481074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.500089] LustreError: 12075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 299.488046] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.488438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.497583] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 299.518147] Lustre: Skipped 2 previous similar messages [ 300.898366] Lustre: server umount lustre-MDT0001 complete [ 304.149482] Lustre: server umount lustre-OST0000 complete [ 307.807987] Lustre: server umount lustre-OST0001 complete [ 313.890988] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 322.926967] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 361.309491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.523470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 361.549600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 361.652572] Lustre: lustre-MDT0000: new disk, initializing [ 361.747285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 361.784782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 366.388539] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 378.422559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.604493] Lustre: 15046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 378.700411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 378.702246] Lustre: Skipped 1 previous similar message [ 378.787130] Lustre: lustre-MDT0001: new disk, initializing [ 378.933846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 378.992294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 379.007648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 384.224729] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 394.222711] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 399.505454] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 406.929544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 407.296541] Lustre: lustre-OST0000: new disk, initializing [ 407.299966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 407.307487] Lustre: 17015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 407.413280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.618639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 408.632932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 408.743271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 413.693141] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 423.809525] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 429.169608] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 429.360833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 434.717451] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 435.048437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 439.269937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 439.275301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 439.287048] Lustre: Skipped 1 previous similar message [ 439.307044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 439.790534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 443.592116] Lustre: server umount lustre-OST0000 complete [ 450.019760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.031014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 450.032197] Lustre: Skipped 2 previous similar messages [ 450.047011] Lustre: Skipped 2 previous similar messages [ 459.235897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.252328] Lustre: Skipped 3 previous similar messages [ 462.304789] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 462.529232] Lustre: server umount lustre-MDT0000 complete [ 464.354888] LustreError: 15059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.371594] LustreError: 15059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 466.880309] LustreError: 15039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784935663 with bad export cookie 9377232003654875889 [ 466.895231] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 467.276554] Lustre: server umount lustre-MDT0001 complete [ 472.362063] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:27:46 (1784935666) [ 475.292186] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 476.739347] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:27:52 (1784935672) [ 483.701936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.139719] LustreError: 19426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.200798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.002655] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 489.443148] LustreError: 19427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.283330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.223640] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 510.447670] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 517.155815] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 524.935463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 531.560896] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 538.493586] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 548.635281] Lustre: server umount lustre-OST0000 complete [ 552.928773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.930678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.942255] Lustre: Skipped 1 previous similar message [ 562.242724] Lustre: server umount lustre-MDT0000 complete [ 563.192793] LustreError: 19427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.232896] LustreError: 19427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 566.496624] LustreError: 19406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784935763 with bad export cookie 9377232003654876771 [ 566.499860] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.508270] LustreError: 19406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 577.245714] Lustre: server umount lustre-MDT0001 complete [ 589.603334] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 592.186269] Key type lgssc unregistered [ 592.449818] LNet: 22960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.454613] LNetError: 22960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.488909] LNet: Removed LNI 192.168.201.136@tcp [ 593.452163] Key type .llcrypt unregistered [ 593.453899] Key type ._llcrypt unregistered [ 615.207640] Key type ._llcrypt registered [ 615.214115] Key type .llcrypt registered [ 615.335797] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 616.084758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.098639] alg: No test for adler32 (adler32-zlib) [ 617.110829] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 617.301176] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 618.943194] Key type lgssc registered [ 619.976179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 627.367126] loop: module loaded [ 627.435816] loop0: detected capacity change from 0 to 204800000 [ 627.657326] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.674579] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.909948] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.119593] loop0: detected capacity change from 0 to 204800000 [ 633.291610] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.309843] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.339851] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.138737] loop0: detected capacity change from 0 to 204800000 [ 643.533662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 643.555611] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.816562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 644.851841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 644.924941] Lustre: lustre-MDT0000: new disk, initializing [ 644.994533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.008628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 649.484388] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 659.415467] loop1: detected capacity change from 0 to 204800000 [ 665.068354] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 665.475980] Lustre: lustre-OST07c6: new disk, initializing [ 665.481254] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 665.485649] Lustre: Skipped 1 previous similar message [ 665.493266] Lustre: 26149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 665.554525] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 672.368750] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 674.961744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 674.988513] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 675.038185] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 677.746551] Lustre: 25135:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 677.757348] Lustre: 25135:0:(osd_handler.c:2091:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 677.769243] Lustre: 25135:0:(osd_handler.c:2098:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 677.780429] Lustre: 25135:0:(osd_handler.c:2108:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 677.789053] Lustre: 25135:0:(osd_handler.c:2115:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 677.800616] Lustre: 25135:0:(osd_handler.c:2122:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 695.270596] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 695.292710] Lustre: 25136:0:(osd_handler.c:2091:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 695.299984] Lustre: 25136:0:(osd_handler.c:2098:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 695.312757] Lustre: 25136:0:(osd_handler.c:2108:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 695.339738] Lustre: 25136:0:(osd_handler.c:2115:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 695.353752] Lustre: 25136:0:(osd_handler.c:2122:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 700.218242] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 700.227657] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 700.232223] Lustre: 25136:0:(osd_handler.c:2091:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 700.239395] Lustre: 25136:0:(osd_handler.c:2091:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.246242] Lustre: 25136:0:(osd_handler.c:2098:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 700.254792] Lustre: 25136:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.262728] Lustre: 25136:0:(osd_handler.c:2108:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 700.269878] Lustre: 25136:0:(osd_handler.c:2108:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.274927] Lustre: 25136:0:(osd_handler.c:2115:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 700.281467] Lustre: 25136:0:(osd_handler.c:2115:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.294919] Lustre: 25136:0:(osd_handler.c:2122:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 700.302741] Lustre: 25136:0:(osd_handler.c:2122:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.551083] Lustre: 25137:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 703.560333] Lustre: 25137:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 703.565785] Lustre: 25137:0:(osd_handler.c:2091:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 703.571318] Lustre: 25137:0:(osd_handler.c:2091:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.574819] Lustre: 25137:0:(osd_handler.c:2098:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.581128] Lustre: 25137:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.586150] Lustre: 25137:0:(osd_handler.c:2108:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 703.592509] Lustre: 25137:0:(osd_handler.c:2108:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.597661] Lustre: 25137:0:(osd_handler.c:2115:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 703.605987] Lustre: 25137:0:(osd_handler.c:2115:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.615785] Lustre: 25137:0:(osd_handler.c:2122:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.621024] Lustre: 25137:0:(osd_handler.c:2122:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.405721] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 708.413573] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 708.426165] Lustre: 25136:0:(osd_handler.c:2091:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 708.431909] Lustre: 25136:0:(osd_handler.c:2091:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.438240] Lustre: 25136:0:(osd_handler.c:2098:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 708.454944] Lustre: 25136:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.465557] Lustre: 25136:0:(osd_handler.c:2108:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 708.474798] Lustre: 25136:0:(osd_handler.c:2108:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.483116] Lustre: 25136:0:(osd_handler.c:2115:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 708.494087] Lustre: 25136:0:(osd_handler.c:2115:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.503615] Lustre: 25136:0:(osd_handler.c:2122:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.511906] Lustre: 25136:0:(osd_handler.c:2122:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.338205] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 720.356159] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 720.370334] Lustre: 25136:0:(osd_handler.c:2091:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 720.381441] Lustre: 25136:0:(osd_handler.c:2091:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.396471] Lustre: 25136:0:(osd_handler.c:2098:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.403833] Lustre: 25136:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.414565] Lustre: 25136:0:(osd_handler.c:2108:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 720.419913] Lustre: 25136:0:(osd_handler.c:2108:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.425173] Lustre: 25136:0:(osd_handler.c:2115:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 720.433644] Lustre: 25136:0:(osd_handler.c:2115:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.442453] Lustre: 25136:0:(osd_handler.c:2122:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.448066] Lustre: 25136:0:(osd_handler.c:2122:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.069926] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.094226] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 774.502374] Lustre: server umount lustre-OST07c6 complete [ 786.577221] Lustre: server umount lustre-MDT0000 complete [ 807.749679] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 811.339301] Key type lgssc unregistered [ 811.727640] LNet: 28709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.737127] LNetError: 28709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.758606] LNet: Removed LNI 192.168.201.136@tcp [ 812.576388] Key type .llcrypt unregistered [ 812.578559] Key type ._llcrypt unregistered [ 831.099777] Key type ._llcrypt registered [ 831.105987] Key type .llcrypt registered [ 831.236281] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 832.306351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 832.471559] alg: No test for adler32 (adler32-zlib) [ 833.629534] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 833.832699] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 835.480329] Key type lgssc registered [ 836.847750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 844.803912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 844.822929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.208223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.789956] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 856.544364] LustreError: 29981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.558630] LustreError: 29981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 859.704510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.166995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.034224] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 873.961256] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 880.560260] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 887.061946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 887.320335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.449984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 893.853872] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 900.578416] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 920.035641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.038528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 920.091337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 920.098199] Lustre: Skipped 1 previous similar message [ 925.086148] Lustre: server umount lustre-OST0000 complete [ 930.276621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.278155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.298546] Lustre: Skipped 2 previous similar messages [ 930.313719] Lustre: Skipped 1 previous similar message [ 935.395966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.416303] Lustre: Skipped 1 previous similar message [ 935.604430] Lustre: server umount lustre-MDT0000 complete [ 939.943770] LustreError: 29961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936136 with bad export cookie 17104744187055543391 [ 939.953580] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.499743] Lustre: server umount lustre-MDT0001 complete [ 953.753619] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 957.534812] Key type lgssc unregistered [ 957.949419] LNet: 33726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.958401] LNetError: 33726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.978721] LNet: Removed LNI 192.168.201.136@tcp [ 958.897334] Key type .llcrypt unregistered [ 958.899323] Key type ._llcrypt unregistered [ 977.077206] Key type ._llcrypt registered [ 977.079090] Key type .llcrypt registered [ 977.169234] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 978.168906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.383528] alg: No test for adler32 (adler32-zlib) [ 979.665898] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 980.169250] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 981.943183] Key type lgssc registered [ 983.342138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.312679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 991.360576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.998412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.432807] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1002.472880] LustreError: 34996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.495337] LustreError: 34996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1007.352752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.651820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1012.602231] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1021.205777] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1027.975736] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1034.902459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1035.190610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.241698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1041.210925] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1049.842947] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1068.003791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.006964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1068.029340] Lustre: Skipped 1 previous similar message [ 1068.044239] Lustre: Skipped 1 previous similar message [ 1070.118292] Lustre: server umount lustre-OST0000 complete [ 1074.144017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1074.146189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.157044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1078.241709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1078.241914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.239535] Lustre: server umount lustre-MDT0000 complete [ 1083.367778] LustreError: 34992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.395337] LustreError: 34992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1083.677888] LustreError: 34976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936280 with bad export cookie 10804721397412538957 [ 1083.691653] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1083.965734] Lustre: server umount lustre-MDT0001 complete [ 1094.877503] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1096.954479] Key type lgssc unregistered [ 1097.196817] LNet: 38552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.201599] LNetError: 38552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.219898] LNet: Removed LNI 192.168.201.136@tcp [ 1098.069969] Key type .llcrypt unregistered [ 1098.071769] Key type ._llcrypt unregistered [ 1120.479373] Key type ._llcrypt registered [ 1120.484737] Key type .llcrypt registered [ 1120.606753] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1121.395842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1121.435917] alg: No test for adler32 (adler32-zlib) [ 1122.744651] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 1123.019694] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1124.824288] Key type lgssc registered [ 1126.064778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1134.315632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1134.345461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.843865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.363392] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1145.318730] LustreError: 39971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.332150] LustreError: 39971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1150.435218] LustreError: 39970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.717435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.120369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1156.491163] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1164.474791] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1170.608500] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1177.546250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1177.901395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.001823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1184.521696] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1192.815718] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1205.261104] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1205.729300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.755261] Lustre: Skipped 2 previous similar messages [ 1205.773906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1205.777413] Lustre: Skipped 2 previous similar messages [ 1208.554786] Lustre: server umount lustre-MDT0000 complete [ 1210.363164] LustreError: 39965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.619499] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936409 with bad export cookie 2084389469660738882 [ 1212.633333] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1212.635196] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1215.477691] LustreError: 42461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.488928] LustreError: 42461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1215.512177] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [ 1215.969044] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.024214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1218.038278] Lustre: Skipped 1 previous similar message [ 1219.121953] Lustre: server umount lustre-MDT0001 complete [ 1234.218451] Lustre: server umount lustre-OST0000 complete [ 1249.494720] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1252.135782] Key type lgssc unregistered [ 1252.409546] LNet: 43831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.419546] LNetError: 43831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.448814] LNet: Removed LNI 192.168.201.136@tcp [ 1253.164567] Key type .llcrypt unregistered [ 1253.171345] Key type ._llcrypt unregistered [ 1276.470823] Key type ._llcrypt registered [ 1276.473560] Key type .llcrypt registered [ 1276.545650] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1277.861284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1277.891084] alg: No test for adler32 (adler32-zlib) [ 1278.881418] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 1279.170918] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1280.831995] Key type lgssc registered [ 1281.832911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1289.647695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1289.675660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.288969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.816704] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1301.473957] LustreError: 45248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.493720] LustreError: 45248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1304.340640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.887508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1309.428297] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1317.439203] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1323.467588] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1330.793160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1331.093293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.283375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1337.939767] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1346.499935] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1359.372825] Lustre: lustre-OST0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1359.841780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1359.854123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.865860] Lustre: Skipped 1 previous similar message [ 1359.881667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1359.892271] Lustre: Skipped 1 previous similar message [ 1363.139547] Lustre: server umount lustre-OST0000 complete [ 1381.856247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1381.860837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.881143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.416586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.421858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.459578] Lustre: Skipped 1 previous similar message [ 1387.987107] Lustre: server umount lustre-MDT0000 complete [ 1389.540171] LustreError: 45244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.560569] LustreError: 45244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1393.100501] LustreError: 45228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936590 with bad export cookie 1910620174424698792 [ 1393.119493] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1393.127251] LustreError: 45228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1399.934197] Lustre: server umount lustre-MDT0001 complete [ 1412.857955] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1417.282336] Key type lgssc unregistered [ 1417.778655] LNet: 48960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.796475] LNetError: 48960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.824393] LNet: Removed LNI 192.168.201.136@tcp [ 1419.031781] Key type .llcrypt unregistered [ 1419.035314] Key type ._llcrypt unregistered [ 1446.878189] Key type ._llcrypt registered [ 1446.884460] Key type .llcrypt registered [ 1446.990239] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1448.210176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.352797] alg: No test for adler32 (adler32-zlib) [ 1449.616654] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 1449.973774] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1451.695766] Key type lgssc registered [ 1452.849734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1461.223922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1461.257875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.916400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.594601] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1472.485814] LustreError: 50377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1472.516364] LustreError: 50377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1477.614104] LustreError: 50376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.204387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.574521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1483.818457] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1492.784546] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1499.012173] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1505.887657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1506.275484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.324362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1513.590753] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1521.288109] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1529.369916] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1533.185376] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1545.389786] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-2419h45m39s [ 1547.188818] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1550.309864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1550.315538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.349442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1551.393605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1551.399629] Lustre: Skipped 4 previous similar messages [ 1553.426296] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1556.120353] Lustre: server umount lustre-MDT0000 complete [ 1557.479399] LustreError: 50377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1557.493866] LustreError: 50377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1559.661576] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1560.567532] LustreError: 50373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1566.720790] LustreError: 50373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1566.756934] LustreError: 50373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1567.063063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1567.201587] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1567.541900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.630963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1568.595164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1572.992832] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1576.711199] Lustre: DEBUG MARKER: Wait for df (36809) ... [ 1578.686047] Lustre: DEBUG MARKER: done [ 1583.073247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1583.084171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.109619] Lustre: Skipped 2 previous similar messages [ 1583.119353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1583.126150] Lustre: Skipped 1 previous similar message [ 1584.100441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1589.240923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1589.244575] Lustre: Skipped 2 previous similar messages [ 1596.895200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1597.076340] Lustre: server umount lustre-OST0000 complete [ 1604.576596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1604.577522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.593334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1604.593343] Lustre: Skipped 2 previous similar messages [ 1604.632021] Lustre: Skipped 1 previous similar message [ 1607.763656] Lustre: server umount lustre-MDT0000 complete [ 1611.487650] LustreError: 53003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784936808 with bad export cookie 11251124054372734625 [ 1611.504364] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1611.504482] LustreError: 53003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1611.766277] Lustre: server umount lustre-MDT0001 complete [ 1622.597783] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1625.247441] Key type lgssc unregistered [ 1625.547057] LNet: 55304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.551503] LNetError: 55304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.564084] LNet: Removed LNI 192.168.201.136@tcp [ 1626.279320] Key type .llcrypt unregistered [ 1626.280953] Key type ._llcrypt unregistered [ 1666.545735] Key type ._llcrypt registered [ 1666.547448] Key type .llcrypt registered [ 1666.667510] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1668.237521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1668.328274] alg: No test for adler32 (adler32-zlib) [ 1669.388971] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 1669.630959] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1671.303163] Key type lgssc registered [ 1672.358504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1680.260598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1680.278582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.563461] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1681.593417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1681.797815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.359604] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1694.264418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.356545] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1694.416900] Lustre: 57516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1694.636379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1699.931996] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1707.913516] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1713.416842] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1721.323448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1721.622511] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1721.809629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.874087] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1731.724988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1736.370681] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1744.972853] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-2419h48m59s [ 1747.480993] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1755.195559] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1758.118473] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1758.200328] LustreError: 57522:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1778.743116] Lustre: lustre-MDT0000: Client dfe4c9ce-a4aa-485b-b851-7c3244288b44 (at 192.168.201.36@tcp) reconnecting [ 1798.295786] LustreError: 57522:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1801.186269] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1804.565541] Lustre: DEBUG MARKER: done [ 1813.476257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1813.482192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.501524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1813.995315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.002923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1814.009123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.918677] Lustre: server umount lustre-OST0000 complete [ 1822.691984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1822.695058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.714992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.826430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.839526] Lustre: Skipped 4 previous similar messages [ 1829.297845] Lustre: server umount lustre-MDT0000 complete [ 1834.559950] LustreError: 59375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937031 with bad export cookie 5754331046302654015 [ 1834.577358] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1835.138643] Lustre: server umount lustre-MDT0001 complete [ 1846.356898] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1849.452685] Key type lgssc unregistered [ 1849.924689] LNet: 61845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.936760] LNetError: 61845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.989201] LNet: Removed LNI 192.168.201.136@tcp [ 1851.787339] Key type .llcrypt unregistered [ 1851.790082] Key type ._llcrypt unregistered [ 1901.129670] Key type ._llcrypt registered [ 1901.133907] Key type .llcrypt registered [ 1901.430678] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1903.808335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.833520] alg: No test for adler32 (adler32-zlib) [ 1904.935930] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 1905.286704] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1906.991755] Key type lgssc registered [ 1908.382136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1912.541629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1912.561905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.936213] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1913.998970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1914.308830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.298927] Lustre: Failing over lustre-MDT0000 [ 1916.030631] Lustre: server umount lustre-MDT0000 complete [ 1922.770334] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:51:57 (1784937117) [ 1930.991639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.519840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.307393] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1947.586390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.701715] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1947.746921] Lustre: 64644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1948.041218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.720494] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1962.928068] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1969.449558] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1977.303124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1977.722402] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1978.013475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.316250] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1985.823507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1994.225714] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2000.515646] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2009.569597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2009.577069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.595714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.622978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.623310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.636310] Lustre: Skipped 1 previous similar message [ 2011.645785] Lustre: Skipped 1 previous similar message [ 2013.464226] Lustre: server umount lustre-MDT0000 complete [ 2017.070619] LustreError: 64638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937213 with bad export cookie 12858325415055533190 [ 2017.089411] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.094288] LustreError: 64638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2017.430976] Lustre: server umount lustre-MDT0001 complete [ 2021.196606] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2026.003179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2035.496488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.215416] Lustre: 63568:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784937218/real 1784937218] req@000000002353bd98 x1871642206710400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784937234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2037.250850] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.240904] Lustre: 63569:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784937219/real 1784937219] req@00000000cfc05f9d x1871642206710528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784937235 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2042.336601] Lustre: 63570:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784937223/real 1784937223] req@000000004262a3e7 x1871642206710656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784937239 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2042.435675] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb271f2d6b2c32d69 [ 2042.451824] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [ 2042.864718] LustreError: 66495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.880723] LustreError: 66495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2043.044924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.214275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2046.053898] LustreError: 66496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.079950] LustreError: 66496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2048.104883] LustreError: 68419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.135929] LustreError: 68419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2049.119493] Lustre: 63568:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784937230/real 1784937230] req@00000000df9b2fdc x1871642206711040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784937246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2049.161256] Lustre: 63568:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2049.447394] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2053.606938] LustreError: 68420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.739274] LustreError: 66496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.756455] LustreError: 66496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2059.384128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.904212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2065.395225] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2065.958913] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2066.488280] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2075.981643] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.401631] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2092.009214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.012691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.016211] Lustre: Skipped 2 previous similar messages [ 2092.025008] Lustre: Skipped 3 previous similar messages [ 2097.124493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.134676] Lustre: Skipped 2 previous similar messages [ 2097.685841] Lustre: server umount lustre-MDT0000 complete [ 2101.777087] LustreError: 64639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937298 with bad export cookie 12858325415055535465 [ 2101.780744] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2101.790146] LustreError: 64639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2102.156959] Lustre: server umount lustre-MDT0001 complete [ 2109.324773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2118.370339] Lustre: 63568:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784937299/real 1784937299] req@0000000044afe27f x1871642206741248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784937315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2118.409516] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.308226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.631088] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb271f2d6b2c332cc [ 2126.640224] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [ 2127.040142] LustreError: 66496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.122303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.175363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2128.095662] Lustre: 63567:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784937308/real 1784937308] req@00000000d62ba61b x1871642206741760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784937324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.129760] Lustre: 63567:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2133.043521] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2142.308936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.671901] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2147.752728] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2155.766840] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2162.833428] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2173.422540] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.425620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.425899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2173.429358] Lustre: Skipped 1 previous similar message [ 2173.478246] Lustre: Skipped 2 previous similar messages [ 2179.404880] Lustre: server umount lustre-MDT0000 complete [ 2183.650167] LustreError: 71105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.671245] LustreError: 71105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2183.923990] LustreError: 65381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937380 with bad export cookie 12858325415055536844 [ 2183.926865] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2183.931737] LustreError: 65381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2184.226070] Lustre: server umount lustre-MDT0001 complete [ 2192.715206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2197.103947] Lustre: DEBUG MARKER: files compared the same [ 2211.426115] Lustre: server umount lustre-OST0000 complete [ 2233.475475] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2236.658595] Key type lgssc unregistered [ 2237.049766] LNet: 74382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.067550] LNetError: 74382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.081522] LNet: Removed LNI 192.168.201.136@tcp [ 2237.987567] Key type .llcrypt unregistered [ 2237.989893] Key type ._llcrypt unregistered [ 2261.634696] Key type ._llcrypt registered [ 2261.636645] Key type .llcrypt registered [ 2261.745708] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2262.468645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2262.496150] alg: No test for adler32 (adler32-zlib) [ 2263.622795] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 2263.814290] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2265.511334] Key type lgssc registered [ 2266.746705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2276.119865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2276.155058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.805541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.930660] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug +malloc all [ 2288.106163] LustreError: 75792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.142308] LustreError: 75792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2293.223618] LustreError: 75793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.808903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.359724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2299.399456] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug +malloc all [ 2308.513286] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2316.337512] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2325.953295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.427877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.594399] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug +malloc all [ 2343.266663] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2348.981116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2355.173272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2355.176381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.188467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.264190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2359.271212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.276797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.298676] Lustre: Skipped 1 previous similar message [ 2360.453585] Lustre: server umount lustre-OST0000 complete [ 2364.383950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.398053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2364.406671] Lustre: Skipped 1 previous similar message [ 2369.504838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2369.514147] Lustre: Skipped 1 previous similar message [ 2370.532923] Lustre: server umount lustre-MDT0000 complete [ 2374.472099] LustreError: 75774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937571 with bad export cookie 10453418855733253801 [ 2374.488878] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.672271] Lustre: server umount lustre-MDT0001 complete [ 2390.904681] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2395.239366] Key type lgssc unregistered [ 2395.608947] LNet: 79325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.625684] LNetError: 79325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2395.644564] LNet: Removed LNI 192.168.201.136@tcp [ 2397.153533] Key type .llcrypt unregistered [ 2397.157844] Key type ._llcrypt unregistered [ 2424.137517] Key type ._llcrypt registered [ 2424.139771] Key type .llcrypt registered [ 2424.267050] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2425.803655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2425.916637] alg: No test for adler32 (adler32-zlib) [ 2427.003244] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 2427.260688] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2428.983585] Key type lgssc registered [ 2430.253824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2439.150405] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2439.186637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2454.943880] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2460.070261] LustreError: 80715:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.136@tcp: failed processing log, type 4: rc = -110 [ 2486.880382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.582850] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2503.065683] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2512.843612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.948715] Lustre: *** cfs_fail_loc=706, val=0*** [ 2513.262041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.318853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2518.499446] LustreError: 80740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.559117] LustreError: 80740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2519.881811] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2521.642303] LustreError: 81814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.701525] LustreError: 81816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.731230] LustreError: 81816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2531.089098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.724659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2537.909840] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2547.783930] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2554.569975] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2557.415132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.423427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2557.430401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2557.440255] Lustre: Skipped 1 previous similar message [ 2562.533785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.546038] Lustre: Skipped 1 previous similar message [ 2563.396566] Lustre: server umount lustre-OST0000 complete [ 2572.771840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.773412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2572.774818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.774828] Lustre: Skipped 1 previous similar message [ 2572.798592] Lustre: Skipped 1 previous similar message [ 2574.974429] Lustre: server umount lustre-MDT0000 complete [ 2577.899259] LustreError: 81819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.942152] LustreError: 81819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2579.723659] LustreError: 80722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784937776 with bad export cookie 12349197209598959175 [ 2579.741292] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2579.745958] LustreError: 80722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2580.208429] Lustre: server umount lustre-MDT0001 complete [ 2591.985185] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2594.833509] Key type lgssc unregistered [ 2595.117812] LNet: 84346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2595.127895] LNetError: 84346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2595.159581] LNet: Removed LNI 192.168.201.136@tcp [ 2595.948416] Key type .llcrypt unregistered [ 2595.949748] Key type ._llcrypt unregistered [ 2621.988578] Key type ._llcrypt registered [ 2621.991684] Key type .llcrypt registered [ 2622.092810] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2623.149178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2623.219120] alg: No test for adler32 (adler32-zlib) [ 2624.505125] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 2624.765130] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2626.567204] Key type lgssc registered [ 2628.026826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2636.678969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2636.696426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.430314] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2656.229729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.116327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2662.368509] LustreError: 86451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.393294] LustreError: 86451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2663.346705] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2666.464371] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.488071] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2671.588782] LustreError: 86454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.625192] LustreError: 86454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2676.704242] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.722976] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2676.741533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2677.263114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.315198] LustreError: 86454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.365815] LustreError: 86454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2686.965784] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2691.557700] LustreError: 86449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.574788] LustreError: 86449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2697.341904] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2697.353123] Lustre: MGS: Client 38f3c320-732e-4ec6-bac5-10e046c1bd88 (at 0@lo) reconnecting [ 2697.375934] LustreError: 86465:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2697.928836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.065441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2702.179466] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2718.179071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2718.191205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.211611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2718.695571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.376139] Lustre: server umount lustre-OST0000 complete [ 2728.933412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.940162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2728.941294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2728.961672] Lustre: Skipped 1 previous similar message [ 2728.965196] Lustre: Skipped 1 previous similar message [ 2731.244833] Lustre: server umount lustre-MDT0000 complete [ 2734.048172] LustreError: 86454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.062953] LustreError: 86454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2735.587709] Lustre: server umount lustre-MDT0001 complete [ 2739.690833] Lustre: server umount MGS complete [ 2751.556545] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:05:46 (1784937946) [ 2783.657734] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 2794.027453] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2832.265498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.243652] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2847.991420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.165508] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2848.194617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2848.252662] Lustre: lustre-MDT0001: new disk, initializing [ 2848.310941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2848.333472] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2848.340045] Lustre: 93621:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2848.349284] LustreError: 93621:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2853.071846] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2868.326396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.648783] Lustre: lustre-OST0000: new disk, initializing [ 2868.653382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2868.659843] Lustre: 94387:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2868.746928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.724249] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2879.591025] Lustre: 94765:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000076efb86f x1871642960636288/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784938092 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2890.168459] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2898.083637] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.091908] Lustre: MGS: Client 5b83a150-6ca3-4476-936b-a0ce7869d1db (at 0@lo) reconnecting [ 2898.151540] Lustre: 95434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2898.165048] LustreError: 93606:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2898.180068] LustreError: 93606:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2898.231397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2898.355104] Lustre: lustre-MDT0000: new disk, initializing [ 2898.552746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.585758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2898.616631] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2899.686473] Lustre: 95459:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000bae458a3 x1871642960643200/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1784938112 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2903.199804] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2904.802745] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2907.116122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2907.130707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2908.787757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2908.798228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2908.876262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2934.753155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2934.758500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.770122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.776854] Lustre: Skipped 1 previous similar message [ 2937.839790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2937.854152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.270727] Lustre: server umount lustre-OST0000 complete [ 2948.067195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.067531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2948.100276] Lustre: Skipped 1 previous similar message [ 2948.117676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.126968] Lustre: Skipped 2 previous similar messages [ 2950.775574] Lustre: server umount lustre-MDT0000 complete [ 2953.184645] LustreError: 93592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.216056] LustreError: 93592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2961.735737] Lustre: server umount lustre-MDT0001 complete [ 2965.712994] Lustre: server umount MGS complete [ 2977.412626] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:09:32 (1784938172) [ 2995.931154] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2998.983048] Key type lgssc unregistered [ 2999.301578] LNet: 97958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2999.309564] LNetError: 97958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3000.380524] LNet: Removed LNI 192.168.201.136@tcp [ 3001.383393] Key type .llcrypt unregistered [ 3001.389809] Key type ._llcrypt unregistered [ 3019.712254] Key type ._llcrypt registered [ 3019.714490] Key type .llcrypt registered [ 3019.818937] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3021.033399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3021.340290] alg: No test for adler32 (adler32-zlib) [ 3022.332687] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 3022.537997] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3024.207991] Key type lgssc registered [ 3025.579975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3034.485776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3034.517118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.071933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.589367] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3045.346561] LustreError: 99182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.369556] LustreError: 99182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3050.471135] LustreError: 99181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3051.521981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.114913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3057.460319] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3065.915522] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3072.695565] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3080.673242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3081.182687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3088.989224] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3098.536172] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3105.781652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3121.121562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3121.131328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.144360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3121.153060] Lustre: Skipped 1 previous similar message [ 3125.162522] Lustre: server umount lustre-OST0000 complete [ 3131.372664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.384433] Lustre: Skipped 2 previous similar messages [ 3131.386082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3131.395415] Lustre: Skipped 1 previous similar message [ 3136.510062] Lustre: server umount lustre-MDT0000 complete [ 3136.535624] LustreError: 99176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.559257] LustreError: 99176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3140.637018] LustreError: 99162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938337 with bad export cookie 17171450358458070155 [ 3140.644429] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3140.651624] LustreError: 99162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3141.197367] Lustre: server umount lustre-MDT0001 complete [ 3151.820305] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3154.851988] Key type lgssc unregistered [ 3155.145952] LNet: 102735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3155.151706] LNetError: 102735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3155.176120] LNet: Removed LNI 192.168.201.136@tcp [ 3156.059547] Key type .llcrypt unregistered [ 3156.061441] Key type ._llcrypt unregistered [ 3173.204824] Key type ._llcrypt registered [ 3173.210478] Key type .llcrypt registered [ 3173.388954] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3174.869788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3174.971096] alg: No test for adler32 (adler32-zlib) [ 3176.078791] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 3176.333748] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3177.999353] Key type lgssc registered [ 3178.951393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3186.041931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3186.049172] LustreError: 103809:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3186.186073] LustreError: 103819:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3186.197558] LustreError: 103809:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4858 [ 3186.205555] LustreError: 103809:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3186.229123] LustreError: 103809:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3186.263717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.056258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.452486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.891296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.873599] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3208.701663] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing lsmod [ 3211.691212] LustreError: 105012:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3211.826781] LustreError: 105026:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3211.839539] LustreError: 105012:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4856 [ 3211.848131] LustreError: 105012:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3211.856963] LustreError: 105012:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3211.864785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3212.213555] LustreError: 105037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.254625] LustreError: 105037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3212.524335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.378436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3217.383264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.404700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.891369] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.904272] Lustre: Skipped 1 previous similar message [ 3217.906480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.920037] Lustre: Skipped 1 previous similar message [ 3221.769848] Lustre: server umount lustre-MDT0000 complete [ 3223.011747] LustreError: 103854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.027591] LustreError: 103854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3224.051801] LustreError: 103858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.074802] LustreError: 103858:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3225.264599] LustreError: 103839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938422 with bad export cookie 6511765133186100148 [ 3225.278682] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3225.288853] LustreError: 103839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3226.207302] LustreError: 104289:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000490263fc x1871643539428096/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3226.372253] Lustre: server umount lustre-MDT0001 complete [ 3240.670137] Lustre: server umount lustre-OST0000 complete [ 3251.672655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3252.036572] LustreError: 106090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.103686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.226229] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3257.316187] LustreError: 106091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.655524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.201403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3271.265386] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3281.273477] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3288.297668] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3297.163370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.370588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3303.328268] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3312.020080] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3321.968167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3331.554410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3331.569345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.591919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3332.579029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3334.581171] Lustre: server umount lustre-OST0000 complete [ 3341.799513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3341.805214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.828133] Lustre: Skipped 1 previous similar message [ 3341.849341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3341.858155] Lustre: Skipped 2 previous similar messages [ 3347.938127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3347.947428] Lustre: Skipped 3 previous similar messages [ 3353.055145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3353.311059] Lustre: server umount lustre-MDT0000 complete [ 3357.065632] LustreError: 106070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938553 with bad export cookie 6511765133186100862 [ 3357.071642] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.076794] LustreError: 106070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3357.389361] Lustre: server umount lustre-MDT0001 complete [ 3370.560961] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3374.541859] Key type lgssc unregistered [ 3375.010764] LNet: 109648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3375.025704] LNetError: 109648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3375.112153] LNet: Removed LNI 192.168.201.136@tcp [ 3376.920268] Key type .llcrypt unregistered [ 3376.922338] Key type ._llcrypt unregistered [ 3407.828627] Key type ._llcrypt registered [ 3407.830446] Key type .llcrypt registered [ 3407.927663] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3409.667029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3409.737226] alg: No test for adler32 (adler32-zlib) [ 3410.859455] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 3411.145024] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3412.831359] Key type lgssc registered [ 3414.337716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3421.976527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3421.996024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.395237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.729024] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3433.962877] LustreError: 111070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.987923] LustreError: 111070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3438.083603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.483690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3444.016181] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3453.142992] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3459.652510] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3467.320561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.758102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.817903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3475.974276] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3483.932968] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3490.855184] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3495.214422] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3503.606235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.611576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3503.619847] Lustre: Skipped 1 previous similar message [ 3503.620795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.601674] Lustre: server umount lustre-OST0000 complete [ 3513.826182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.849820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.859532] Lustre: Skipped 2 previous similar messages [ 3517.694257] Lustre: server umount lustre-MDT0000 complete [ 3518.950120] LustreError: 111800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.979714] LustreError: 111800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3522.004882] LustreError: 111051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938718 with bad export cookie 258411310257739111 [ 3522.005339] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3522.031348] LustreError: 111051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3522.453733] Lustre: server umount lustre-MDT0001 complete [ 3535.743209] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3539.000402] Key type lgssc unregistered [ 3539.313600] LNet: 114708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3539.327393] LNetError: 114708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3540.393033] LNet: Removed LNI 192.168.201.136@tcp [ 3541.282948] Key type .llcrypt unregistered [ 3541.284560] Key type ._llcrypt unregistered [ 3560.586786] Key type ._llcrypt registered [ 3560.590820] Key type .llcrypt registered [ 3560.748761] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3562.418561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3562.503057] alg: No test for adler32 (adler32-zlib) [ 3563.681159] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 3564.071288] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3565.807209] Key type lgssc registered [ 3567.514972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3576.428804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3576.460445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.227048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.403325] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3588.580139] LustreError: 115933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.618536] LustreError: 115933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3593.718232] LustreError: 115934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3596.583496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.118535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3601.865182] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3611.144639] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3617.892188] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3625.829878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3626.183489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.909089] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3640.513359] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3648.696895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3659.244475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3659.252089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.264606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3659.274905] Lustre: Skipped 1 previous similar message [ 3664.353204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3664.363489] Lustre: Skipped 1 previous similar message [ 3664.681689] Lustre: server umount lustre-OST0000 complete [ 3668.967316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3668.985351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.008707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3669.011261] Lustre: Skipped 1 previous similar message [ 3674.593278] LustreError: 115934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.011719] Lustre: server umount lustre-MDT0000 complete [ 3679.329015] LustreError: 115913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938876 with bad export cookie 2834755380862560683 [ 3679.360859] LustreError: 115913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3679.361684] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3679.878606] Lustre: server umount lustre-MDT0001 complete [ 3690.863819] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3693.174885] Key type lgssc unregistered [ 3693.438667] LNet: 119489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3693.448773] LNetError: 119489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3693.475711] LNet: Removed LNI 192.168.201.136@tcp [ 3694.433275] Key type .llcrypt unregistered [ 3694.437751] Key type ._llcrypt unregistered [ 3722.602994] Key type ._llcrypt registered [ 3722.605960] Key type .llcrypt registered [ 3722.758586] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3724.019198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3724.080145] alg: No test for adler32 (adler32-zlib) [ 3725.238913] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 3725.675992] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3727.376380] Key type lgssc registered [ 3729.506315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3740.898509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3740.951587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.932152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.987517] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3752.435712] LustreError: 120908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.463474] LustreError: 120908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3757.541931] LustreError: 120909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.850497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.385154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3766.032464] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3776.114300] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3784.826832] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3794.090826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3794.650945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3802.874552] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3812.549718] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3813.128413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3820.886036] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3824.917961] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3824.925826] Lustre: Skipped 2 previous similar messages [ 3829.801357] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3829.806666] Lustre: Skipped 2 previous similar messages [ 3830.437246] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3830.505742] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3840.155180] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3840.160319] Lustre: Skipped 2 previous similar messages [ 3849.889741] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.36@tcp 192.168.0.[2,10]@tcp [ 3854.928795] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.36/24@tcp in log lustre-MDT0000 [ 3854.933748] Lustre: Skipped 2 previous similar messages [ 3859.565140] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.36/24@tcp [ 3859.574948] Lustre: Skipped 1 previous similar message [ 3872.742969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3872.763235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.796846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3874.799529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3874.800984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3874.807671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3874.838152] Lustre: Skipped 1 previous similar message [ 3878.180712] Lustre: server umount lustre-OST0000 complete [ 3883.489391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3883.508285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.520592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.145119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.147568] Lustre: Skipped 3 previous similar messages [ 3895.264716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.311462] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3897.827378] Lustre: server umount lustre-MDT0000 complete [ 3899.370538] LustreError: 121868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.411368] LustreError: 121868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3902.822744] LustreError: 120888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939099 with bad export cookie 501772343758498336 [ 3902.832779] LustreError: 120888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3902.834875] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3903.348660] Lustre: server umount lustre-MDT0001 complete [ 3916.404161] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3919.864490] Key type lgssc unregistered [ 3920.129065] LNet: 125800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3920.143847] LNetError: 125800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3920.162884] LNet: Removed LNI 192.168.201.136@tcp [ 3921.410319] Key type .llcrypt unregistered [ 3921.413102] Key type ._llcrypt unregistered [ 3955.109285] Key type ._llcrypt registered [ 3955.113835] Key type .llcrypt registered [ 3955.212806] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3956.247909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3956.354814] alg: No test for adler32 (adler32-zlib) [ 3957.458251] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 3957.749259] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3959.503201] Key type lgssc registered [ 3960.907700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3970.035577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3970.062188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3971.702847] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3971.753684] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.36/24@tcp [ 3971.765563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.841496] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3981.282391] LustreError: 127366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.336150] LustreError: 127366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3986.408902] LustreError: 127365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.840645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3988.262837] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3988.301527] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.36/24@tcp [ 3988.353599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3994.172160] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4003.473984] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4010.416329] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4017.597393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4017.939709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4026.699283] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4037.344244] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4041.600445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4053.986129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4054.001649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.019886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4057.068094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4057.076650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4057.084405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4057.093693] Lustre: Skipped 1 previous similar message [ 4060.186190] Lustre: server umount lustre-OST0000 complete [ 4065.251699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4065.263363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.277011] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.297697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.299032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.333166] Lustre: Skipped 1 previous similar message [ 4070.462500] Lustre: server umount lustre-MDT0000 complete [ 4072.421063] LustreError: 127365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4072.447243] LustreError: 127365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4074.770935] LustreError: 127345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939271 with bad export cookie 1824868185669515877 [ 4074.778688] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4074.780693] LustreError: 127345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4075.044760] Lustre: server umount lustre-MDT0001 complete [ 4087.340865] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4090.952498] Key type lgssc unregistered [ 4091.175541] LNet: 130975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4091.185142] LNetError: 130975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4091.194573] LNet: Removed LNI 192.168.201.136@tcp [ 4091.898075] Key type .llcrypt unregistered [ 4091.909929] Key type ._llcrypt unregistered [ 4122.145906] Key type ._llcrypt registered [ 4122.149444] Key type .llcrypt registered [ 4122.258491] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4137.507339] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4139.241243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4139.287672] alg: No test for adler32 (adler32-zlib) [ 4140.535491] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 4140.975141] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4142.743174] Key type lgssc registered [ 4144.432968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4185.821660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4185.879293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.115943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4187.136817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4187.289875] Lustre: lustre-MDT0000: new disk, initializing [ 4187.415091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.433601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4191.941467] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4203.743033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.854284] Lustre: 134376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4203.903082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4203.907652] Lustre: Skipped 1 previous similar message [ 4203.968562] Lustre: lustre-MDT0001: new disk, initializing [ 4204.044395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4204.076385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4204.081484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4207.354615] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4217.265541] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4223.597996] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4231.071446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4231.313945] Lustre: lustre-OST0000: new disk, initializing [ 4231.318191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4231.322445] Lustre: 136344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4231.393969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4237.944785] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4238.435936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4238.453515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4238.538973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4247.759256] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4252.070134] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4252.342840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4255.970388] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4256.106657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4259.296838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4259.301566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.313315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4259.327067] Lustre: Skipped 1 previous similar message [ 4264.238823] Lustre: server umount lustre-OST0000 complete [ 4269.537237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.545981] Lustre: Skipped 1 previous similar message [ 4269.547773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.554090] Lustre: Skipped 2 previous similar messages [ 4274.660136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4274.668174] Lustre: Skipped 1 previous similar message [ 4279.777426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4279.785056] Lustre: Skipped 1 previous similar message [ 4283.360293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4283.656231] Lustre: server umount lustre-MDT0000 complete [ 4287.840326] LustreError: 134370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939484 with bad export cookie 735832190828408853 [ 4287.851346] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4287.853146] LustreError: 134370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4288.269646] Lustre: server umount lustre-MDT0001 complete [ 4299.072882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.518045] LustreError: 138563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.532645] LustreError: 138563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4299.631188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.868494] LustreError: 138564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.593773] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4310.008962] LustreError: 138563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.108163] LustreError: 138564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.272807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.792574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4322.124248] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4330.887744] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4337.577379] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4345.466406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4346.084282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.921344] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4363.564207] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4370.011674] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4370.322973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4375.419268] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4375.675354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4386.426222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4386.585884] Lustre: lustre-OST0001: new disk, initializing [ 4386.589605] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4386.594371] Lustre: 141934:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4386.674807] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4392.271682] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4393.141664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4393.158401] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4393.306418] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4400.360786] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4400.559299] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4404.901113] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4405.119846] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4410.690351] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff956b91993800.ost_server_uuid 50 [ 4412.523173] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff956b91993800.ost_server_uuid in FULL state after 0 sec [ 4424.162743] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4424.170183] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4424.178339] LustreError: Skipped 2 previous similar messages [ 4424.179027] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4424.187547] Lustre: Skipped 1 previous similar message [ 4424.201946] Lustre: Skipped 1 previous similar message [ 4427.359159] Lustre: server umount lustre-OST0001 complete [ 4429.300891] LustreError: 140384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.330240] LustreError: 140384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4431.332169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4431.339261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.376579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4433.389419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4433.392016] Lustre: Skipped 1 previous similar message [ 4437.470396] Lustre: server umount lustre-OST0000 complete [ 4444.640780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4444.654664] Lustre: Skipped 2 previous similar messages [ 4449.764056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.779813] Lustre: Skipped 5 previous similar messages [ 4455.391662] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4455.610813] Lustre: server umount lustre-MDT0000 complete [ 4459.358647] LustreError: 138543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939656 with bad export cookie 735832190828409735 [ 4459.367090] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4459.374551] LustreError: 138543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4459.719348] Lustre: server umount lustre-MDT0001 complete [ 4479.395279] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4482.244929] Key type lgssc unregistered [ 4482.537630] LNet: 144938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4482.544810] LNetError: 144938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4483.623074] LNet: Removed LNI 192.168.201.136@tcp [ 4484.476275] Key type .llcrypt unregistered [ 4484.478573] Key type ._llcrypt unregistered [ 4528.915456] Key type ._llcrypt registered [ 4528.917240] Key type .llcrypt registered [ 4529.012934] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4544.854476] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4546.507403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4546.572263] alg: No test for adler32 (adler32-zlib) [ 4547.625907] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 4547.929690] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4549.615189] Key type lgssc registered [ 4551.327644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4595.326371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4595.352505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.669102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4596.754195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4596.821567] Lustre: lustre-MDT0000: new disk, initializing [ 4596.914888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.936734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4601.745909] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4612.382435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4612.507983] Lustre: 149100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4612.573162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4612.579828] Lustre: Skipped 1 previous similar message [ 4612.658402] Lustre: lustre-MDT0001: new disk, initializing [ 4612.763204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4612.800710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4612.818598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4617.985379] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4627.364184] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4633.768783] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4641.939118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4642.126417] Lustre: lustre-OST0000: new disk, initializing [ 4642.129362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4642.134250] Lustre: 151068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4642.204928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.835251] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4649.540745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4649.561885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4649.630439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4657.177941] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4661.428225] Lustre: Failing over lustre-OST0000 [ 4661.563782] Lustre: server umount lustre-OST0000 complete [ 4679.136159] Lustre: 147322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939860/real 1784939860] req@00000000db503000 x1871644977037184/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1784939876 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4679.186554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4679.983312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4680.344948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.378776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4680.391704] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4681.439295] Lustre: 147322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939862/real 1784939862] req@000000001bca608c x1871644977037696/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784939878 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4684.415771] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4686.431262] Lustre: 147325:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939867/real 1784939867] req@00000000cfc05f9d x1871644977039360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784939883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4686.458565] Lustre: 147325:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4686.936889] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4689.631106] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4689.632059] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4691.213293] Lustre: Failing over lustre-MDT0000 [ 4691.733923] Lustre: server umount lustre-MDT0000 complete [ 4692.255630] Lustre: 147325:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939872/real 1784939872] req@00000000511583f0 x1871644977040384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784939888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4692.274451] Lustre: 147325:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4695.015919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4695.031596] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.049990] LustreError: 150711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.056975] Lustre: Skipped 2 previous similar messages [ 4695.079570] LustreError: 150711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4695.612143] LustreError: 149107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.135150] LustreError: 149110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.146480] LustreError: 149110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4705.250735] LustreError: 149107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.273667] LustreError: 149107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4710.367687] Lustre: 147323:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784939891/real 1784939891] req@0000000076efb86f x1871644977050624/t0(0) o400->MGC192.168.201.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784939907 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4710.374425] LustreError: 150711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.394322] Lustre: 147323:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4710.416942] LustreError: 150711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4710.442605] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4711.903234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.612166] LustreError: 151075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.638753] LustreError: 151075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4720.859485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4720.922462] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4721.153921] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4726.261498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4726.269127] Lustre: Skipped 1 previous similar message [ 4726.282086] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4726.308271] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4726.377120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4736.490526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.509181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4736.510981] Lustre: Skipped 1 previous similar message [ 4736.525433] Lustre: Skipped 1 previous similar message [ 4741.621483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4742.738622] Lustre: server umount lustre-OST0000 complete [ 4746.721387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4746.723939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.732528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4746.752809] Lustre: Skipped 1 previous similar message [ 4746.765465] Lustre: Skipped 2 previous similar messages [ 4751.842646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4751.849887] Lustre: Skipped 1 previous similar message [ 4753.055226] Lustre: server umount lustre-MDT0000 complete [ 4756.960826] LustreError: 149106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4756.980382] LustreError: 149106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4757.458241] LustreError: 149895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939954 with bad export cookie 640313810071858796 [ 4757.459196] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4757.465145] LustreError: 149895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4757.745865] Lustre: server umount lustre-MDT0001 complete [ 4769.865703] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4772.433427] Key type lgssc unregistered [ 4772.741039] LNet: 154727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4772.771169] LNetError: 154727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4773.796825] LNet: Removed LNI 192.168.201.136@tcp [ 4774.778740] Key type .llcrypt unregistered [ 4774.781731] Key type ._llcrypt unregistered [ 4810.931548] Key type ._llcrypt registered [ 4810.935361] Key type .llcrypt registered [ 4811.066946] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4812.104669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4812.133211] alg: No test for adler32 (adler32-zlib) [ 4813.280827] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 4813.461135] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4815.095187] Key type lgssc registered [ 4816.300989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4823.611328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4823.623883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4825.027319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.948117] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4834.280074] LustreError: 156258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.302296] LustreError: 156258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4838.927896] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4839.394555] LustreError: 156259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.887652] LustreError: 156288:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4842.893532] LustreError: 156288:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4842.897504] LustreError: 156288:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4843.223160] Lustre: server umount lustre-MDT0000 complete [ 4865.131770] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4879.210926] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4921.556986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4921.854410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4921.895404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4921.983209] Lustre: lustre-MDT0000: new disk, initializing [ 4922.063295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4922.085507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4927.476117] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4937.590568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4937.721879] Lustre: 159593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4937.780204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4937.784092] Lustre: Skipped 1 previous similar message [ 4937.881931] Lustre: lustre-MDT0001: new disk, initializing [ 4938.009856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4938.053014] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4938.076749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4942.666774] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4952.835457] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4959.714705] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4967.559513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4967.838750] Lustre: lustre-OST0000: new disk, initializing [ 4967.843926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4967.854404] Lustre: 161560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4967.957780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4974.642609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4974.657183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4974.823872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4975.061731] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4985.640178] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4991.140426] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4992.856509] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:43:07 (1784940187) [ 5255.647724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5255.650474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.663838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5256.671626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5256.672751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5256.676460] Lustre: Skipped 1 previous similar message [ 5256.692892] Lustre: Skipped 1 previous similar message [ 5261.793961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.799889] Lustre: Skipped 3 previous similar messages [ 5266.399151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5266.636969] Lustre: server umount lustre-MDT0000 complete [ 5266.917361] LustreError: 161878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5266.932777] LustreError: 161878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5270.900475] LustreError: 161559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940467 with bad export cookie 14609398477914265586 [ 5270.905282] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5270.912813] LustreError: 161559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5272.032106] LustreError: 161795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5272.046801] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.094324] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5280.169570] LustreError: 159601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.169823] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5280.195153] LustreError: 159601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5280.237497] Lustre: Skipped 1 previous similar message [ 5285.343453] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5285.681936] Lustre: server umount lustre-MDT0001 complete [ 5306.335215] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5306.461590] Lustre: server umount lustre-OST0000 complete [ 5317.590924] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5328.280574] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5369.440755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.712405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5369.735666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5369.855400] Lustre: lustre-MDT0000: new disk, initializing [ 5369.958854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5369.969957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5375.358104] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5387.538537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.698453] Lustre: 166235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5387.788903] LustreError: 166244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.799416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5387.804574] LustreError: 166244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5387.806013] Lustre: Skipped 1 previous similar message [ 5387.883207] Lustre: lustre-MDT0001: new disk, initializing [ 5387.983487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5388.016836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5388.037171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5393.626265] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5404.928731] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5411.982906] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5419.885225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5420.293815] Lustre: lustre-OST0000: new disk, initializing [ 5420.300232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5420.314328] Lustre: 168200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5420.449922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5427.240028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5427.246571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5427.359720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5427.795545] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5439.324555] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5445.299202] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5445.476838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5450.425226] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5450.737162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5453.288945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5453.293298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.296153] LustreError: Skipped 1 previous similar message [ 5453.296808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5453.296812] Lustre: Skipped 1 previous similar message [ 5453.327169] Lustre: Skipped 1 previous similar message [ 5459.326453] Lustre: server umount lustre-OST0000 complete [ 5465.056676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5465.075115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5473.760472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5473.768028] Lustre: Skipped 7 previous similar messages [ 5478.367218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5478.547317] Lustre: server umount lustre-MDT0000 complete [ 5478.885040] LustreError: 166241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5478.903941] LustreError: 166241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5483.069810] LustreError: 167029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940679 with bad export cookie 14609398477914424416 [ 5483.083976] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5483.086917] LustreError: 167029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5483.656766] Lustre: server umount lustre-MDT0001 complete [ 5493.456448] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:51:27 (1784940687) [ 5515.007820] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5530.171714] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5575.266319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.575943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5575.618535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5575.738912] Lustre: lustre-MDT0000: new disk, initializing [ 5575.830085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.861590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5580.470456] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5591.545398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.699598] Lustre: 172979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5592.049244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5596.867583] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5607.694535] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5614.989479] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5622.554052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5622.812851] Lustre: lustre-OST0000: new disk, initializing [ 5622.815575] Lustre: Skipped 1 previous similar message [ 5622.818977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5622.822049] Lustre: Skipped 2 previous similar messages [ 5622.830537] Lustre: 174957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5628.512320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5628.521731] Lustre: Skipped 1 previous similar message [ 5628.530937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5628.660776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5629.851355] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5640.036201] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5649.376793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5649.378674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5649.389684] LustreError: Skipped 1 previous similar message [ 5649.390394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5649.408479] Lustre: Skipped 2 previous similar messages [ 5655.255360] Lustre: server umount lustre-OST0000 complete [ 5665.334533] Lustre: server umount lustre-MDT0000 complete [ 5669.457972] LustreError: 173778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940866 with bad export cookie 14609398477914425298 [ 5669.464492] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.499061] LustreError: 173778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5670.021381] Lustre: server umount lustre-MDT0001 complete [ 5680.215248] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:54:34 (1784940874) [ 5697.945598] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5710.401814] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5750.798444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.010136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5751.032878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5751.089314] Lustre: lustre-MDT0000: new disk, initializing [ 5751.148231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5751.157342] Lustre: Skipped 2 previous similar messages [ 5751.168251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5756.077469] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5766.780258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.881209] Lustre: 179280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5767.100786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5772.053482] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5781.483114] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5786.726563] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5794.415611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5794.746178] Lustre: 181261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5802.092165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5802.242221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5802.801494] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5813.639411] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5822.953494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5822.954889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.962319] LustreError: Skipped 1 previous similar message [ 5822.966445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.983795] Lustre: Skipped 3 previous similar messages [ 5822.987254] Lustre: Skipped 7 previous similar messages [ 5828.713508] Lustre: server umount lustre-OST0000 complete [ 5838.878944] Lustre: server umount lustre-MDT0000 complete [ 5842.651808] LustreError: 179273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941039 with bad export cookie 14609398477914426383 [ 5842.660598] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5842.664967] LustreError: 179273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5843.245485] Lustre: server umount lustre-MDT0001 complete [ 5855.381767] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 5858.760879] Key type lgssc unregistered [ 5859.185233] LNet: 183117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5859.194544] LNetError: 183117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5859.218983] LNet: Removed LNI 192.168.201.136@tcp [ 5860.432533] Key type .llcrypt unregistered [ 5860.434193] Key type ._llcrypt unregistered [ 5886.393829] Key type ._llcrypt registered [ 5886.396923] Key type .llcrypt registered [ 5886.506652] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5887.683560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5887.957510] alg: No test for adler32 (adler32-zlib) [ 5889.219543] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 5889.509738] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 5891.191250] Key type lgssc registered [ 5892.702531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5900.329733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5900.350444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.699079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5905.826908] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5912.037308] LustreError: 184535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5912.066466] LustreError: 184535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5915.419979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.065655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5921.627881] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5930.637662] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5937.354178] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5945.548712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5946.135510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5947.145029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5954.127393] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5963.195355] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5986.273341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5986.276557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5986.292480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5986.303637] Lustre: Skipped 1 previous similar message [ 5986.313910] Lustre: Skipped 1 previous similar message [ 5990.729439] Lustre: server umount lustre-OST0000 complete [ 5996.517346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.521087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5996.530725] Lustre: Skipped 1 previous similar message [ 5996.544637] Lustre: Skipped 1 previous similar message [ 6001.637088] LustreError: 185263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.664864] Lustre: server umount lustre-MDT0000 complete [ 6005.895686] LustreError: 185271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941202 with bad export cookie 6748453623188116564 [ 6005.902780] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6005.912034] LustreError: 185271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6006.676922] Lustre: server umount lustre-MDT0001 complete [ 6021.589812] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 6024.951897] Key type lgssc unregistered [ 6025.500386] LNet: 188097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6025.519320] LNetError: 188097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6025.552911] LNet: Removed LNI 192.168.201.136@tcp [ 6026.718628] Key type .llcrypt unregistered [ 6026.721683] Key type ._llcrypt unregistered [ 6056.405434] Key type ._llcrypt registered [ 6056.408338] Key type .llcrypt registered [ 6056.569373] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6058.004789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6058.153052] alg: No test for adler32 (adler32-zlib) [ 6059.337596] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 6059.711035] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 6061.501346] Key type lgssc registered [ 6062.680971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6071.485728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6071.498863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.139499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6078.837410] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6083.556270] LustreError: 189529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6083.581229] LustreError: 189529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6088.684170] LustreError: 189530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.201831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.602419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6093.963681] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6104.343755] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6110.314428] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6118.452140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.891611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.160577] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6136.598588] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6143.474919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6148.586590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.592698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6148.654174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6148.663812] Lustre: Skipped 1 previous similar message [ 6151.099596] Lustre: server umount lustre-OST0000 complete [ 6160.490104] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff956b847cd800.ost_server_uuid 50 [ 6162.684303] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6179.299401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.321268] Lustre: Skipped 1 previous similar message [ 6179.324319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6179.328156] Lustre: Skipped 2 previous similar messages [ 6183.138599] Lustre: server umount lustre-MDT0000 complete [ 6184.428685] LustreError: 189524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.456536] LustreError: 189524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6187.752545] LustreError: 189511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941384 with bad export cookie 12658519362909899720 [ 6187.760375] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6187.762662] LustreError: 189511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6188.217795] Lustre: server umount lustre-MDT0001 complete [ 6197.768562] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:03:12 (1784941392) [ 6205.575920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6206.200231] LustreError: 193450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6206.298810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.009211] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6211.558501] LustreError: 193451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.799171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.141642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6226.247776] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6234.394718] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6241.308874] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6248.651645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6249.022760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6252.085993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6255.980660] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6264.158506] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6272.021929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6272.235299] Lustre: lustre-OST0001: new disk, initializing [ 6272.238528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6272.247435] Lustre: 196291:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6272.355904] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6279.457049] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6281.814591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6281.834210] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6281.975211] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6289.370994] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6292.451676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.455321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6292.472413] Lustre: Skipped 2 previous similar messages [ 6292.487124] Lustre: Skipped 1 previous similar message [ 6295.052475] Lustre: lustre-OST0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 6300.168584] Lustre: lustre-OST0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 6300.171208] Lustre: Skipped 2 previous similar messages [ 6306.273739] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6306.349228] Lustre: server umount lustre-OST0000 complete [ 6307.834076] LustreError: 196964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6307.859752] LustreError: 196964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6313.399964] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6313.684626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6318.536628] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6318.780574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6325.755712] LustreError: 196964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.783119] LustreError: 196964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6333.408165] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6333.416606] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.418215] LustreError: Skipped 1 previous similar message [ 6333.419464] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6333.419470] Lustre: Skipped 3 previous similar messages [ 6333.449545] Lustre: Skipped 1 previous similar message [ 6338.268966] Lustre: server umount lustre-OST0001 complete [ 6343.649065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.660863] Lustre: Skipped 1 previous similar message [ 6348.807516] Lustre: server umount lustre-MDT0000 complete [ 6352.699621] LustreError: 193433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941549 with bad export cookie 12658519362909900791 [ 6352.702679] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6352.709911] LustreError: 193433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6353.002997] Lustre: server umount lustre-MDT0001 complete [ 6377.012601] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:06:12 (1784941572) [ 6384.159206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.445748] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6384.497927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6384.702540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6389.782451] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6398.204278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6398.348436] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6398.376127] Lustre: 199735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6404.314930] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6412.654072] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6419.000118] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6426.397513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6426.528652] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6432.955898] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6435.022857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6441.291423] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6448.471368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6448.608687] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6448.888487] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6448.898681] Lustre: Skipped 2 previous similar messages [ 6455.746339] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6462.926412] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6466.025987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6466.030160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6466.035954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6466.044251] Lustre: Skipped 1 previous similar message [ 6466.057550] Lustre: Skipped 4 previous similar messages [ 6471.632052] Lustre: server umount lustre-OST0000 complete [ 6473.733342] LustreError: 201586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.762278] LustreError: 201586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6491.621620] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6491.622953] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6491.648773] LustreError: Skipped 1 previous similar message [ 6492.760353] Lustre: server umount lustre-OST0001 complete [ 6496.225553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6501.856585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6501.858979] Lustre: Skipped 7 previous similar messages [ 6502.442590] Lustre: server umount lustre-MDT0000 complete [ 6505.621133] LustreError: 200490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941702 with bad export cookie 12658519362909902198 [ 6505.625279] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6505.629444] LustreError: 200490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6505.975961] Lustre: server umount lustre-MDT0001 complete [ 6525.308807] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:08:40 (1784941720) [ 6537.908219] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 6545.975684] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6575.535316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6575.778743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6575.806946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6575.880454] Lustre: lustre-MDT0000: new disk, initializing [ 6575.939614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6579.394336] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6587.764156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.847736] Lustre: 207826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6587.884325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6587.888802] Lustre: Skipped 1 previous similar message [ 6587.942855] Lustre: lustre-MDT0001: new disk, initializing [ 6588.018367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6588.025323] Lustre: Skipped 1 previous similar message [ 6588.047188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6588.053787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6591.261917] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6599.854556] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6604.322812] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6609.657274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6609.898467] Lustre: lustre-OST0000: new disk, initializing [ 6609.901139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6609.911833] Lustre: 209794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6615.459592] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6617.131497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6617.142851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6617.234904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6623.316969] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6627.705272] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6627.932378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6631.621236] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6631.774905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6637.537532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6637.556969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6637.574050] Lustre: Skipped 3 previous similar messages [ 6637.580804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6647.263669] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6647.441614] Lustre: server umount lustre-OST0000 complete [ 6665.183137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6665.360524] Lustre: server umount lustre-MDT0000 complete [ 6668.567048] LustreError: 207820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941865 with bad export cookie 12658519362909903759 [ 6668.575060] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6668.579894] LustreError: 207820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6668.921436] Lustre: server umount lustre-MDT0001 complete [ 6676.306611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6676.609379] LustreError: 212012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.621837] LustreError: 212012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6680.795223] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6687.697950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.580440] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6697.698048] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6702.475383] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6708.483182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6714.314050] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6720.228211] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6723.963041] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6724.157049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6728.018485] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6728.172811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6731.748128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6731.751679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.759325] LustreError: Skipped 1 previous similar message [ 6731.769310] Lustre: Skipped 3 previous similar messages [ 6735.671953] Lustre: server umount lustre-OST0000 complete [ 6740.476990] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6743.478941] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6770.209900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6771.560903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6771.572718] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6771.574906] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6771.590543] Lustre: Skipped 1 previous similar message [ 6774.986313] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6780.167688] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6791.498901] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 9 sec [ 6796.089674] Lustre: server umount lustre-OST0000 complete [ 6800.865192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6800.872089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6800.873731] Lustre: Skipped 12 previous similar messages [ 6808.675536] Lustre: server umount lustre-MDT0000 complete [ 6811.757596] LustreError: 211994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942008 with bad export cookie 12658519362909904641 [ 6811.758891] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6811.764845] LustreError: 211994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6821.492305] Lustre: server umount lustre-MDT0001 complete [ 6828.161443] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:13:43 (1784942023) [ 6832.919867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.306136] LustreError: 217999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6833.321248] LustreError: 217999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6836.564897] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6841.805041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.359199] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6849.437765] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6853.596850] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6858.740530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6859.002348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6859.007875] Lustre: Skipped 7 previous similar messages [ 6864.111922] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6869.186199] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6872.710217] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6879.196939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 6882.644798] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6882.828324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6887.906741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6888.031587] Lustre: lustre-OST0001: new disk, initializing [ 6888.033906] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6888.037614] Lustre: 221408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6892.611621] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6896.197449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6896.204790] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6896.262543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6900.054142] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6903.354868] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6903.515658] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6906.431760] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6906.563979] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6911.968596] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6911.972862] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6911.979448] LustreError: Skipped 1 previous similar message [ 6911.994863] Lustre: Skipped 3 previous similar messages [ 6922.207161] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6922.294668] Lustre: server umount lustre-OST0001 complete [ 6926.707253] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6926.914930] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6929.570753] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6929.699482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6956.246512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6960.693468] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6961.648225] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6961.679046] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6961.694317] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6966.065334] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6967.208820] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6974.760648] Lustre: server umount lustre-OST0001 complete [ 6994.689496] Lustre: server umount lustre-OST0000 complete [ 7003.832835] Lustre: server umount lustre-MDT0000 complete [ 7006.363385] LustreError: 217980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942203 with bad export cookie 12658519362909905607 [ 7006.368159] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7006.373519] LustreError: 217980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7012.667342] Lustre: server umount lustre-MDT0001 complete [ 7026.846454] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:17:02 (1784942222) [ 7030.716194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.890292] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7030.911276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7033.923676] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7039.621584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7039.697668] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7039.717635] Lustre: 226648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7042.751575] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7047.904755] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7051.773361] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7056.907659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7057.021340] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7061.426240] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7066.848974] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7073.698712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7073.773787] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7077.824181] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7082.781627] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7083.949076] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7087.043228] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7087.192617] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7089.999392] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7090.102579] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7093.397429] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff956b847ca000.ost_server_uuid 50 [ 7094.476392] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff956b847ca000.ost_server_uuid in FULL state after 0 sec [ 7095.110315] Lustre: Permanently deactivating lustre-OST0001 [ 7095.118699] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7096.964543] Lustre: Permanently reactivating lustre-OST0001 [ 7100.398511] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7100.401469] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7100.406985] LustreError: Skipped 5 previous similar messages [ 7100.414905] Lustre: Skipped 21 previous similar messages [ 7105.243949] Lustre: server umount lustre-OST0001 complete [ 7105.506142] LustreError: 228490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.527246] LustreError: 228490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [ 7121.887628] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7121.982713] Lustre: server umount lustre-OST0000 complete [ 7130.448568] Lustre: server umount lustre-MDT0000 complete [ 7132.879816] LustreError: 229516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942329 with bad export cookie 12658519362909906895 [ 7132.880618] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7132.888028] LustreError: 229516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7133.295499] Lustre: server umount lustre-MDT0001 complete [ 7150.598288] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:19:06 (1784942346) [ 7156.186285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.399292] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7156.420345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7156.425843] Lustre: Skipped 5 previous similar messages [ 7159.528223] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7166.071839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7166.165804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7166.177728] Lustre: 233338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7169.460440] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7175.226705] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7179.802030] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7184.783045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7184.958442] Lustre: Permanently deactivating lustre-OST0000 [ 7184.968360] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7189.485159] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7194.948691] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7195.128026] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7195.173904] LustreError: 188829:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006b91a6a9 x1871646562892544/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7200.092790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7200.145603] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7200.147855] Lustre: Skipped 1 previous similar message [ 7201.658324] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7204.532525] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7209.812159] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7211.000619] Lustre: Permanently reactivating lustre-OST0000 [ 7211.010928] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7211.015552] Lustre: Skipped 2 previous similar messages [ 7211.495368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7211.503369] Lustre: Skipped 12 previous similar messages [ 7211.506788] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7211.512890] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7211.525771] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7211.528423] Lustre: Skipped 1 previous similar message [ 7211.546460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7228.909995] Lustre: server umount lustre-OST0001 complete [ 7237.517568] Lustre: server umount lustre-OST0000 complete [ 7245.777311] Lustre: server umount lustre-MDT0000 complete [ 7247.788526] LustreError: 233330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942444 with bad export cookie 12658519362909908547 [ 7247.792559] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7247.793981] LustreError: 233330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7247.986968] Lustre: server umount lustre-MDT0001 complete [ 7254.901396] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7256.645857] Key type lgssc unregistered [ 7256.851227] LNet: 238212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7256.856409] LNetError: 238212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7256.869268] LNet: Removed LNI 192.168.201.136@tcp [ 7257.445175] Key type .llcrypt unregistered [ 7257.451867] Key type ._llcrypt unregistered [ 7272.648722] Key type ._llcrypt registered [ 7272.651280] Key type .llcrypt registered [ 7272.728995] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7273.311087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7273.333508] alg: No test for adler32 (adler32-zlib) [ 7274.268757] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 7274.417781] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7276.039165] Key type lgssc registered [ 7276.734537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7282.308772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7282.324838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7283.573863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.371291] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7292.781283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7292.885605] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7292.903739] Lustre: 239645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7292.916223] Lustre: Permanently deactivating lustre-MDT0001 [ 7292.925221] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7293.171205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7296.918891] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7296.931111] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7301.627506] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7305.162456] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7310.041764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7310.275896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.600525] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7318.382819] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7318.516071] LustreError: 241498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7318.524391] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7318.524882] LustreError: 241498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7322.977268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7323.138051] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7327.215089] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7328.248705] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7329.263085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7329.271955] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7332.193292] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7334.373737] Lustre: Permanently reactivating lustre-MDT0001 [ 7334.385459] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7334.388569] Lustre: Skipped 2 previous similar messages [ 7339.028722] Lustre: lustre-MDT0001: Client bd7d0459-b3f4-49dd-b849-6613a1f31f7a (at 192.168.201.36@tcp) reconnecting [ 7343.374869] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7344.228798] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7344.689081] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.701970] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7344.707156] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7344.714957] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7348.330259] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7348.441691] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7351.524127] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7351.686094] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7353.404778] Lustre: Permanently deactivating lustre-MDT0001 [ 7353.422061] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7353.424956] Lustre: Skipped 2 previous similar messages [ 7354.008527] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7368.161070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.170592] Lustre: Skipped 3 previous similar messages [ 7368.171780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7368.179497] Lustre: Skipped 3 previous similar messages [ 7373.006562] Lustre: server umount lustre-MDT0000 complete [ 7373.283787] LustreError: 242924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.294501] LustreError: 242924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7374.968319] LustreError: 240411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942571 with bad export cookie 3404348505295649705 [ 7374.969995] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7374.974313] LustreError: 240411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7375.128813] Lustre: server umount lustre-MDT0001 complete [ 7383.187296] Lustre: server umount lustre-OST0000 complete [ 7389.868447] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:23:05 (1784942585) [ 7397.067315] Lustre: server umount lustre-OST0001 complete [ 7400.716946] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 7406.221424] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7425.311909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7425.428978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7425.432949] Lustre: Skipped 2 previous similar messages [ 7425.445500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7425.496687] Lustre: lustre-MDT0000: new disk, initializing [ 7425.527836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7425.539454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7427.506531] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7432.812583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.870052] Lustre: 247906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7432.908532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7432.910270] Lustre: Skipped 1 previous similar message [ 7432.952299] Lustre: lustre-MDT0001: new disk, initializing [ 7433.023646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7433.032659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7434.712589] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7440.355443] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7443.199748] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7446.463736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.626934] Lustre: lustre-OST0000: new disk, initializing [ 7446.629588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7446.634208] Lustre: 249873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7446.682695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7446.685297] Lustre: Skipped 1 previous similar message [ 7449.890615] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7451.653381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7451.656985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7451.697799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7455.431983] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7456.628327] LustreError: 247911:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7462.155113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7462.252217] Lustre: lustre-OST0001: new disk, initializing [ 7462.255113] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7462.259712] Lustre: 250985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7465.385449] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7466.655198] LustreError: 247911:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7466.670276] LustreError: 250640:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7470.819533] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7471.582448] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7471.633166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7471.638944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7471.686816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7476.767238] LustreError: 250640:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7478.789658] Lustre: lustre-OST0001: Not available for connect from 192.168.201.36@tcp (stopping) [ 7481.832580] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.833029] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7481.849799] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7481.852650] LustreError: Skipped 1 previous similar message [ 7481.863182] Lustre: Skipped 1 previous similar message [ 7483.895620] Lustre: lustre-OST0001: Not available for connect from 192.168.201.36@tcp (stopping) [ 7489.014450] Lustre: lustre-OST0001: Not available for connect from 192.168.201.36@tcp (stopping) [ 7489.017960] Lustre: Skipped 2 previous similar messages [ 7492.063113] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7492.133721] Lustre: server umount lustre-OST0001 complete [ 7497.188810] LustreError: 249857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.203559] LustreError: 249857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7502.311235] LustreError: 249859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7502.331751] LustreError: 249859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7507.427291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7507.430033] LustreError: 249859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.436690] Lustre: Skipped 1 previous similar message [ 7507.438067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7507.453880] LustreError: 249859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7507.455643] Lustre: Skipped 3 previous similar messages [ 7511.590264] Lustre: server umount lustre-OST0000 complete [ 7515.105070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7515.115386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.128709] Lustre: Skipped 1 previous similar message [ 7520.008955] Lustre: server umount lustre-MDT0000 complete [ 7522.200769] LustreError: 248700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942719 with bad export cookie 3404348505295652526 [ 7522.202079] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7522.209202] LustreError: 248700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7528.412966] Lustre: server umount lustre-MDT0001 complete [ 7534.373972] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7536.004527] Key type lgssc unregistered [ 7536.182502] LNet: 253020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7536.187109] LNetError: 253020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7536.201267] LNet: Removed LNI 192.168.201.136@tcp [ 7536.808166] Key type .llcrypt unregistered [ 7536.811963] Key type ._llcrypt unregistered [ 7561.332696] Key type ._llcrypt registered [ 7561.333680] Key type .llcrypt registered [ 7561.379107] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7562.009190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7562.038036] alg: No test for adler32 (adler32-zlib) [ 7562.934908] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 7563.057261] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7564.655782] Key type lgssc registered [ 7565.215261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7569.631807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7569.643173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.802350] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7570.820567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7570.930529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7573.287029] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7578.742140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7578.815651] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7578.853594] Lustre: 255203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7579.002987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7581.489945] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7586.057412] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7589.098360] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7592.693752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7592.797233] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7592.899429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7595.906747] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7599.917614] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7601.726660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7647.716359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7647.717147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.719696] LustreError: Skipped 1 previous similar message [ 7647.720848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7647.727073] Lustre: Skipped 1 previous similar message [ 7647.740853] Lustre: Skipped 1 previous similar message [ 7652.533195] Lustre: server umount lustre-OST0000 complete [ 7654.660290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7661.959166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7662.107638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7666.612960] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7667.268807] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7667.295070] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7667.306494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7675.089831] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7724.011796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7724.020877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7724.056220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7727.272514] Lustre: server umount lustre-OST0000 complete [ 7732.706107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7732.720969] LustreError: Skipped 1 previous similar message [ 7732.724519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.743191] Lustre: Skipped 1 previous similar message [ 7732.751932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7732.756726] Lustre: Skipped 1 previous similar message [ 7737.166615] Lustre: server umount lustre-MDT0000 complete [ 7740.553449] LustreError: 255194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942937 with bad export cookie 2532081876952368819 [ 7740.559126] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7740.564064] LustreError: 255194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7740.866440] Lustre: server umount lustre-MDT0001 complete [ 7749.819570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7752.260101] Key type lgssc unregistered [ 7752.558774] LNet: 260621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7752.570132] LNetError: 260621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7752.600677] LNet: Removed LNI 192.168.201.136@tcp [ 7753.454767] Key type .llcrypt unregistered [ 7753.456532] Key type ._llcrypt unregistered [ 7771.199528] Key type ._llcrypt registered [ 7771.201516] Key type .llcrypt registered [ 7771.282463] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7772.492087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7772.548872] alg: No test for adler32 (adler32-zlib) [ 7773.602959] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 7773.786104] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7775.447175] Key type lgssc registered [ 7776.564580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7782.863498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7782.882869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.199713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7787.805473] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7794.657733] LustreError: 262018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.669204] LustreError: 262018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7796.073026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7796.394357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7800.324808] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7806.538797] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7811.299671] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7817.040281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.335082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7823.510947] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7829.633614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7830.045396] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7850.466907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7850.470857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7850.484678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7850.492865] Lustre: Skipped 1 previous similar message [ 7850.505384] Lustre: Skipped 1 previous similar message [ 7855.119912] Lustre: server umount lustre-OST0000 complete [ 7860.705049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.713681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7860.716101] Lustre: Skipped 1 previous similar message [ 7864.924435] Lustre: server umount lustre-MDT0000 complete [ 7865.827789] LustreError: 263103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7865.846944] LustreError: 263103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7867.979027] LustreError: 261998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943064 with bad export cookie 1409240362431076332 [ 7867.987450] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7867.993380] LustreError: 261998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7868.401973] Lustre: server umount lustre-MDT0001 complete [ 7878.286503] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7880.582203] Key type lgssc unregistered [ 7880.901860] LNet: 266321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7880.917774] LNetError: 266321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7880.943121] LNet: Removed LNI 192.168.201.136@tcp [ 7881.847866] Key type .llcrypt unregistered [ 7881.851728] Key type ._llcrypt unregistered [ 7894.818915] Key type ._llcrypt registered [ 7894.819964] Key type .llcrypt registered [ 7894.910397] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7896.018244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7896.039479] alg: No test for adler32 (adler32-zlib) [ 7897.125370] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 7897.453175] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7899.119258] Key type lgssc registered [ 7900.242533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7905.949492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7905.962876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7907.267844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.638856] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7917.539481] LustreError: 267522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.566933] LustreError: 267522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7917.616551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.962690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7921.923515] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7928.573595] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7933.117790] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7938.573491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7938.868573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7944.684961] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7951.195874] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7952.178424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7967.722143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7967.725569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7967.725920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7967.746033] Lustre: Skipped 1 previous similar message [ 7967.768129] Lustre: Skipped 1 previous similar message [ 7971.483926] Lustre: server umount lustre-OST0000 complete [ 7977.955673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7977.962124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7977.962152] Lustre: Skipped 1 previous similar message [ 7977.974047] Lustre: Skipped 1 previous similar message [ 7980.817301] Lustre: server umount lustre-MDT0000 complete [ 7983.073778] LustreError: 267521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7983.097115] LustreError: 267521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7984.293894] LustreError: 269366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943181 with bad export cookie 7949321262766635487 [ 7984.296545] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7984.305500] LustreError: 269366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7984.619366] Lustre: server umount lustre-MDT0001 complete [ 7993.944888] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7996.002403] Key type lgssc unregistered [ 7996.320582] LNet: 271201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7996.324909] LNetError: 271201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7996.342600] LNet: Removed LNI 192.168.201.136@tcp [ 7996.999882] Key type .llcrypt unregistered [ 7997.002043] Key type ._llcrypt unregistered [ 8018.186711] Key type ._llcrypt registered [ 8018.190611] Key type .llcrypt registered [ 8018.343363] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8019.452314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8019.631389] alg: No test for adler32 (adler32-zlib) [ 8020.739885] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 8021.009465] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8022.703470] Key type lgssc registered [ 8024.009793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8032.315355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8032.330743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8033.742514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8037.489359] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8044.007618] LustreError: 272617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8044.025404] LustreError: 272617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8044.599500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.864428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8048.927815] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8055.070750] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8059.870304] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8065.798512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8066.061512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.420228] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8075.880448] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8078.396075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8098.792062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8098.792862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8098.804186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8098.813358] Lustre: Skipped 1 previous similar message [ 8103.906253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8103.915618] Lustre: Skipped 2 previous similar messages [ 8104.160846] Lustre: server umount lustre-OST0000 complete [ 8109.024519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8109.026455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8109.037871] Lustre: Skipped 1 previous similar message [ 8109.047937] Lustre: Skipped 1 previous similar message [ 8113.445716] Lustre: server umount lustre-MDT0000 complete [ 8114.146794] LustreError: 273670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.163556] LustreError: 273670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8116.166528] LustreError: 272599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943313 with bad export cookie 16738345898852115861 [ 8116.171962] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8116.176802] LustreError: 272599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8116.398190] Lustre: server umount lustre-MDT0001 complete [ 8124.892574] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8126.919090] Key type lgssc unregistered [ 8127.085607] LNet: 276921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8127.089580] LNetError: 276921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8127.114450] LNet: Removed LNI 192.168.201.136@tcp [ 8127.762639] Key type .llcrypt unregistered [ 8127.764456] Key type ._llcrypt unregistered [ 8141.392609] Key type ._llcrypt registered [ 8141.394529] Key type .llcrypt registered [ 8141.490707] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8142.139582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8142.161554] alg: No test for adler32 (adler32-zlib) [ 8143.155258] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 8143.333802] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8144.959225] Key type lgssc registered [ 8145.644592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8151.503087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8151.551259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.993119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8156.590612] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8162.274859] LustreError: 278136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8162.294510] LustreError: 278136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8162.978204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8163.253344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8167.187375] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8173.209738] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8177.793646] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8183.351782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8183.658041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.377437] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8194.946488] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8198.001301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8208.353129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8208.359189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.376233] Lustre: Skipped 1 previous similar message [ 8208.382131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8208.386954] Lustre: Skipped 1 previous similar message [ 8213.461604] Lustre: server umount lustre-OST0000 complete [ 8218.595926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8218.605895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8218.607134] Lustre: Skipped 1 previous similar message [ 8218.617595] Lustre: Skipped 1 previous similar message [ 8222.380238] Lustre: server umount lustre-MDT0000 complete [ 8223.714847] LustreError: 278145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8223.738643] LustreError: 278145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8225.387559] LustreError: 278101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943422 with bad export cookie 3026750359784332623 [ 8225.394063] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8225.633377] Lustre: server umount lustre-MDT0001 complete [ 8233.158814] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8234.813737] Key type lgssc unregistered [ 8235.006089] LNet: 281832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8235.013057] LNetError: 281832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8235.033074] LNet: Removed LNI 192.168.201.136@tcp [ 8235.633215] Key type .llcrypt unregistered [ 8235.635681] Key type ._llcrypt unregistered [ 8243.597501] Key type ._llcrypt registered [ 8243.599315] Key type .llcrypt registered [ 8243.686161] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8256.149711] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8266.805439] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8267.438354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8267.452283] alg: No test for adler32 (adler32-zlib) [ 8268.389231] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 8268.587295] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8270.215812] Key type lgssc registered [ 8271.053771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8291.729392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8291.747080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8292.900672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8292.914649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8292.959263] Lustre: lustre-MDT0000: new disk, initializing [ 8293.002751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8293.013698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8295.302755] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8302.577293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8302.643603] Lustre: 285360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8302.676903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8302.679771] Lustre: Skipped 1 previous similar message [ 8302.726206] Lustre: lustre-MDT0001: new disk, initializing [ 8302.786722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8302.817323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8302.825276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8305.346812] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8312.035614] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8315.768458] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8319.867690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8320.091828] Lustre: lustre-OST0000: new disk, initializing [ 8320.096170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8320.102437] Lustre: 287329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8320.165159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8323.802241] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8326.169887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8326.174816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8326.262024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8330.734556] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8333.567849] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8333.735515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8336.483446] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8336.612361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8341.480811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8341.481280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8341.493453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8341.511728] Lustre: Skipped 1 previous similar message [ 8346.595598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8346.599404] Lustre: Skipped 1 previous similar message [ 8351.719293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8351.731209] Lustre: Skipped 1 previous similar message [ 8352.223164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8352.330679] Lustre: server umount lustre-OST0000 complete [ 8356.833277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.838076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8356.853435] Lustre: Skipped 2 previous similar messages [ 8361.207944] Lustre: server umount lustre-MDT0000 complete [ 8364.133133] LustreError: 285352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943561 with bad export cookie 16179333674445536704 [ 8364.135819] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8364.141333] LustreError: 285352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8364.380619] Lustre: server umount lustre-MDT0001 complete [ 8370.290317] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:39:25 (1784943565) [ 8375.373686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.738817] LustreError: 289742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.752544] LustreError: 289742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8375.787535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8379.163590] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8380.901169] LustreError: 289743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8385.272448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.530524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8387.952287] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8392.494143] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8396.322677] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8400.832892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8401.077888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8405.253846] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8410.153425] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8417.766096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8417.768730] LustreError: Skipped 1 previous similar message [ 8417.771430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8417.779207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8417.782923] Lustre: Skipped 1 previous similar message [ 8420.834280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.061580] Lustre: server umount lustre-OST0000 complete [ 8425.954467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8425.966822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8425.968524] Lustre: Skipped 1 previous similar message [ 8425.977737] Lustre: Skipped 3 previous similar messages [ 8427.805901] Lustre: server umount lustre-MDT0000 complete [ 8429.778024] LustreError: 289724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943626 with bad export cookie 16179333674445537593 [ 8429.778467] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8429.784133] LustreError: 289724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8430.046892] Lustre: server umount lustre-MDT0001 complete [ 8436.466464] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8438.047544] Key type lgssc unregistered [ 8438.221426] LNet: 293274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8438.224835] LNetError: 293274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8438.235697] LNet: Removed LNI 192.168.201.136@tcp [ 8438.675369] Key type .llcrypt unregistered [ 8438.677399] Key type ._llcrypt unregistered [ 8455.678617] Key type ._llcrypt registered [ 8455.687763] Key type .llcrypt registered [ 8455.793442] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8465.031600] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8465.803746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8465.820637] alg: No test for adler32 (adler32-zlib) [ 8466.728491] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 8466.911444] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8468.519165] Key type lgssc registered [ 8469.167477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8496.872822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8496.896355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8498.044641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8498.066200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8498.116634] Lustre: lustre-MDT0000: new disk, initializing [ 8498.168355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8498.174915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8501.069436] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8508.179380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8508.237721] Lustre: 297074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8508.275626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8508.281730] Lustre: Skipped 1 previous similar message [ 8508.343719] Lustre: lustre-MDT0001: new disk, initializing [ 8508.409918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8508.455195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8508.467176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8510.884486] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8517.167168] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8520.118850] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8524.597941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8524.770663] Lustre: lustre-OST2710: new disk, initializing [ 8524.773318] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8524.777319] Lustre: 299039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8524.823177] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8528.110087] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8530.964025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8530.971393] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8531.023650] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8533.921121] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8537.291095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8537.361614] Lustre: lustre-OST03e8: new disk, initializing [ 8537.364248] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8537.367486] Lustre: 300102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8537.415612] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8540.243839] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8540.401689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8540.410994] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8540.461670] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8544.038868] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8547.980921] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8548.091904] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8550.306132] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8550.413914] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8552.577088] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8552.690932] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8555.011373] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8555.112103] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8559.584873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8559.589681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8559.596312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.120939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.121870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.128814] Lustre: Skipped 1 previous similar message [ 8563.148752] Lustre: server umount lustre-MDT0000 complete [ 8565.129450] LustreError: 297067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943762 with bad export cookie 7164230442503647903 [ 8565.133226] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8565.137077] LustreError: 297067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8565.319972] Lustre: server umount lustre-MDT0001 complete [ 8577.491040] Lustre: server umount lustre-OST2710 complete [ 8589.859304] Lustre: server umount lustre-OST03e8 complete [ 8599.772722] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8604.726175] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8624.864236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8624.979114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8624.997225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8625.052255] Lustre: lustre-MDT0000: new disk, initializing [ 8625.107312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8625.131591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8627.584856] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8633.312861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8633.368871] Lustre: 305344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8633.408586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8633.410792] Lustre: Skipped 1 previous similar message [ 8633.413348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8633.419325] Lustre: Skipped 3 previous similar messages [ 8633.454447] Lustre: lustre-MDT0001: new disk, initializing [ 8633.498115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8633.516872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8633.522683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8635.761727] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8641.255525] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8644.240747] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8647.757609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8647.897674] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8647.905756] Lustre: 307314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8650.738937] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8653.294993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8653.300659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8653.327861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8656.696200] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8658.642646] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8658.755371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8660.994163] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8661.105586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8663.522398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8663.523395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.528324] LustreError: Skipped 1 previous similar message [ 8663.532214] Lustre: Skipped 1 previous similar message [ 8663.533740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8663.540384] Lustre: Skipped 1 previous similar message [ 8673.764936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8673.769172] Lustre: Skipped 3 previous similar messages [ 8676.319680] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8676.437146] Lustre: server umount lustre-OST0000 complete [ 8678.880322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8678.893829] Lustre: Skipped 2 previous similar messages [ 8693.215155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8693.299334] Lustre: server umount lustre-MDT0000 complete [ 8695.566144] LustreError: 305337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943892 with bad export cookie 7164230442503649583 [ 8695.570309] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8695.577860] LustreError: 305337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8695.820858] Lustre: server umount lustre-MDT0001 complete [ 8700.501458] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:44:55 (1784943895) [ 8701.405955] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8702.570754] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:44:58 (1784943898) [ 8704.368902] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8719.426700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.560790] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8719.574389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8719.666848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.669019] Lustre: Skipped 1 previous similar message [ 8722.277360] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8727.273382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8727.376511] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8727.391733] Lustre: 311165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8729.792624] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8748.016390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.157169] Lustre: Denying initial registration attempt from nid 192.168.201.136@tcp, specified as failover [ 8748.162515] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8748.166854] LustreError: 312621:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8748.170411] LustreError: 312621:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8748.172180] LustreError: 312621:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8748.203213] Lustre: server umount lustre-OST0000 complete [ 8748.205283] LustreError: 312621:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8753.120202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8753.124417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8753.132976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8753.135502] Lustre: Skipped 6 previous similar messages [ 8758.267950] Lustre: server umount lustre-MDT0000 complete [ 8759.263975] LustreError: 311179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8759.269231] LustreError: 311179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8760.509405] LustreError: 311159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943957 with bad export cookie 7164230442503650465 [ 8760.510208] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8760.515686] LustreError: 311159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8760.719518] Lustre: server umount lustre-MDT0001 complete [ 8766.664353] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8768.023096] Key type lgssc unregistered [ 8768.201547] LNet: 313739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8768.205399] LNetError: 313739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8768.222595] LNet: Removed LNI 192.168.201.136@tcp [ 8768.625170] Key type .llcrypt unregistered [ 8768.626372] Key type ._llcrypt unregistered [ 8775.103246] Key type ._llcrypt registered [ 8775.104881] Key type .llcrypt registered [ 8775.175267] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8775.611226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8775.619687] alg: No test for adler32 (adler32-zlib) [ 8776.568574] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 8776.739333] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8778.367144] Key type lgssc registered [ 8778.981529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8811.042338] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8816.321118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8816.335993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.523570] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8817.533928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8817.616618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.187919] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8825.836788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8825.926593] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8825.940418] Lustre: 316477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8826.093768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8828.951380] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8834.183990] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8837.653791] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8854.953167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8855.062491] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8855.184916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8858.847975] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8863.623253] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8865.760683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8865.762372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.764489] LustreError: Skipped 1 previous similar message [ 8865.770966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8865.777797] Lustre: Skipped 1 previous similar message [ 8870.885054] Lustre: server umount lustre-OST0000 complete [ 8876.000665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8876.001779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8876.006289] Lustre: Skipped 2 previous similar messages [ 8876.022735] Lustre: Skipped 1 previous similar message [ 8881.120993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8881.125879] Lustre: Skipped 1 previous similar message [ 8885.216299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8885.218651] Lustre: Skipped 1 previous similar message [ 8887.263123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8887.337997] Lustre: server umount lustre-MDT0000 complete [ 8889.258915] LustreError: 316471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944086 with bad export cookie 6338868091673248658 [ 8889.259307] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8889.263971] LustreError: 316471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8889.508523] Lustre: server umount lustre-MDT0001 complete [ 8895.195864] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8896.500918] Key type lgssc unregistered [ 8896.666628] LNet: 320053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8896.669571] LNetError: 320053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8896.680427] LNet: Removed LNI 192.168.201.136@tcp [ 8897.081290] Key type .llcrypt unregistered [ 8897.083080] Key type ._llcrypt unregistered [ 8910.542514] Key type ._llcrypt registered [ 8910.544144] Key type .llcrypt registered [ 8910.611988] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8911.232877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8911.249293] alg: No test for adler32 (adler32-zlib) [ 8912.228510] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 8912.369687] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8914.023210] Key type lgssc registered [ 8914.748299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8919.071791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8919.079707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8920.426682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8922.921493] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8927.879467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8928.052669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8930.429097] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8934.667689] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8937.570528] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8943.210594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8943.428319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8946.268768] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8950.130439] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8963.042352] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8963.047122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8963.053290] Lustre: Skipped 2 previous similar messages [ 8963.063161] Lustre: Skipped 2 previous similar messages [ 8967.222352] Lustre: server umount lustre-MDT0000 complete [ 8968.161425] LustreError: 323937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8968.167724] LustreError: 323937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8969.064640] LustreError: 321430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944165 with bad export cookie 2986824377156324057 [ 8969.067286] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8969.071074] LustreError: 321430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8973.280664] LustreError: 322501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8973.280952] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8973.292436] LustreError: 322501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8973.302159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8973.310070] Lustre: Skipped 1 previous similar message [ 8974.304418] LustreError: 322501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8974.304860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8974.317413] Lustre: Skipped 1 previous similar message [ 8975.337748] Lustre: server umount lustre-MDT0001 complete [ 8978.222961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8982.083141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8993.759641] LustreError: 320759:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c621deca x1871649554067456/t0(0) o250->MGC192.168.201.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8993.925913] LustreError: 323279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8993.964483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8993.997380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8995.747889] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8999.072161] LustreError: 325039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.076388] LustreError: 325039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8999.611432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8999.790525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9001.639368] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9003.877902] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9005.209799] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9009.030977] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 9011.493873] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9014.425516] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 9019.363544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9019.371826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9019.373285] Lustre: Skipped 1 previous similar message [ 9019.379042] Lustre: Skipped 1 previous similar message [ 9022.449904] Lustre: server umount lustre-OST0000 complete [ 9025.504071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9025.510950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9025.520091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9029.602676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.864562] Lustre: server umount lustre-MDT0000 complete [ 9032.755432] LustreError: 324230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944229 with bad export cookie 2986824377156331211 [ 9032.757944] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9032.759040] LustreError: 324230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9032.908866] Lustre: server umount lustre-MDT0001 complete [ 9038.634307] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9040.141565] Key type lgssc unregistered [ 9040.299454] LNet: 327708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9040.313802] LNetError: 327708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9040.323457] LNet: Removed LNI 192.168.201.136@tcp [ 9040.747168] Key type .llcrypt unregistered [ 9040.751860] Key type ._llcrypt unregistered [ 9054.232489] Key type ._llcrypt registered [ 9054.233801] Key type .llcrypt registered [ 9054.284994] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9054.909247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9054.930200] alg: No test for adler32 (adler32-zlib) [ 9055.820741] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9055.974592] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9057.599220] Key type lgssc registered [ 9058.143857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9062.110650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9062.118205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9063.354647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.710377] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9069.471914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9069.658650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9071.514282] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9074.604634] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9077.073705] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9078.421533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.427806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9078.430760] Lustre: Skipped 1 previous similar message [ 9079.457476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9079.460085] Lustre: Skipped 1 previous similar message [ 9084.578693] Lustre: server umount lustre-MDT0000 complete [ 9084.896295] LustreError: 329098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9084.903360] LustreError: 329098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9086.077315] LustreError: 330719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944282 with bad export cookie 2760835835183676701 [ 9086.079664] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9086.081317] LustreError: 330719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9086.206261] Lustre: server umount lustre-MDT0001 complete [ 9092.589195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.739460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9092.758822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9092.854110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9094.762266] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9098.048925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.093897] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9098.106501] Lustre: 331686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9098.218789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9099.847704] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9102.905924] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9105.126153] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9110.178587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9110.256644] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9110.261147] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9110.269392] LustreError: 333570:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9110.272423] LustreError: 333570:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9110.277375] LustreError: 333570:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9110.312075] Lustre: server umount lustre-OST0000 complete [ 9110.313849] LustreError: 333570:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9115.683497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9115.773511] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9115.890954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9118.413742] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9121.276778] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9121.306514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9124.215777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9124.252468] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9124.285387] Lustre: lustre-OST0001: new disk, initializing [ 9124.287295] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9124.290520] Lustre: 335108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9124.326929] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9126.467155] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9127.566207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9127.573098] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9127.609617] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9129.291829] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9133.026705] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9133.029303] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9133.031140] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9133.033301] Lustre: Skipped 2 previous similar messages [ 9136.513574] Lustre: server umount lustre-OST0001 complete [ 9138.146229] LustreError: 334036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.146498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9138.147357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9138.147364] Lustre: Skipped 1 previous similar message [ 9138.152389] LustreError: 334036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9138.159403] Lustre: Skipped 1 previous similar message [ 9143.266358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9143.266963] LustreError: 334035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9143.268866] Lustre: Skipped 2 previous similar messages [ 9143.276152] LustreError: 334035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9144.152678] Lustre: server umount lustre-OST0000 complete [ 9148.384922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9148.389580] Lustre: Skipped 1 previous similar message [ 9151.861503] Lustre: server umount lustre-MDT0000 complete [ 9153.505246] LustreError: 331699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9153.512689] LustreError: 331699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9153.607146] LustreError: 332426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944350 with bad export cookie 2760835835183677240 [ 9153.608142] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9153.613182] LustreError: 332426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9153.786850] Lustre: server umount lustre-MDT0001 complete [ 9158.862125] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9160.301454] Key type lgssc unregistered [ 9160.476722] LNet: 337064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9160.482303] LNetError: 337064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9160.495276] LNet: Removed LNI 192.168.201.136@tcp [ 9160.947879] Key type .llcrypt unregistered [ 9160.952633] Key type ._llcrypt unregistered [ 9200.352207] Key type ._llcrypt registered [ 9200.354031] Key type .llcrypt registered [ 9200.409922] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9208.195926] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9208.662437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9208.675059] alg: No test for adler32 (adler32-zlib) [ 9209.560948] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9209.682433] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9211.279119] Key type lgssc registered [ 9211.729616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9228.018585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9228.032873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9229.169684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9229.196454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9229.235290] Lustre: lustre-MDT0000: new disk, initializing [ 9229.276974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9229.288577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9231.170589] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9236.600025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9236.688663] Lustre: 341969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9236.719388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9236.721715] Lustre: Skipped 1 previous similar message [ 9236.763155] Lustre: lustre-MDT0001: new disk, initializing [ 9236.804760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9236.818575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9236.821861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9238.644548] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9243.457543] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9246.099778] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9249.500372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9249.656887] Lustre: lustre-OST0000: new disk, initializing [ 9249.659447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9249.663446] Lustre: 343937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9249.719061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9252.769257] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9257.928095] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9259.526748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9259.533449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9259.568086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9259.976452] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9260.094195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9262.265591] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9262.353971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9264.608455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9264.608574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9264.611847] LustreError: Skipped 1 previous similar message [ 9264.619898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9265.315683] Lustre: server umount lustre-OST0000 complete [ 9267.680465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9267.683379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9267.688721] Lustre: Skipped 1 previous similar message [ 9267.691066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9267.693349] Lustre: Skipped 1 previous similar message [ 9269.728079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9269.728629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9273.081508] Lustre: server umount lustre-MDT0000 complete [ 9274.708520] LustreError: 342763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944471 with bad export cookie 20138436248598776 [ 9274.711561] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9274.715094] LustreError: 342763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9274.868514] Lustre: server umount lustre-MDT0001 complete [ 9279.054633] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:54:34 (1784944474) [ 9282.314852] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:54:38 (1784944478) [ 9285.914896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.155424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9287.915227] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9291.234145] LustreError: 346690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9291.242033] LustreError: 346690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9291.654096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9291.833253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9293.730675] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9297.223953] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9299.946380] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9303.310949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9303.459857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9305.996334] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9309.257556] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9320.517871] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9321.340522] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9322.166732] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9322.900677] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9323.723269] Lustre: Failing over lustre-MDT0000 [ 9323.877771] Lustre: server umount lustre-MDT0000 complete [ 9327.072856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9327.074691] LustreError: 346686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.077021] Lustre: Skipped 2 previous similar messages [ 9327.083113] LustreError: 346686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9332.192620] LustreError: 347428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9332.197306] LustreError: 347428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9337.292111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9337.313467] LustreError: 346689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9337.321844] LustreError: 346689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9337.383668] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9337.482913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9337.502282] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9338.952313] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9342.453193] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9342.954798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9342.979882] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9343.012174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9345.949435] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9346.830717] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9349.348891] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9350.321474] Lustre: Failing over lustre-MDT0000 [ 9350.549372] Lustre: server umount lustre-MDT0000 complete [ 9352.701916] LustreError: 346684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9352.709410] LustreError: 346684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9353.185190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9353.185848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9353.190470] Lustre: Skipped 2 previous similar messages [ 9362.934315] LustreError: 346714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9362.945969] LustreError: 346714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9365.689192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.768783] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9365.987534] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9368.051559] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9368.577037] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9371.112688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9371.115145] Lustre: Skipped 2 previous similar messages [ 9371.125682] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9371.147825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9374.733231] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9375.683842] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9381.345186] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.345979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9381.347669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9381.347674] Lustre: Skipped 1 previous similar message [ 9381.349804] Lustre: Skipped 1 previous similar message [ 9384.867194] Lustre: server umount lustre-MDT0000 complete [ 9386.469130] LustreError: 350288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9386.478899] LustreError: 350288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9386.922324] LustreError: 346671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944583 with bad export cookie 20138436248601457 [ 9386.929410] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9386.931266] LustreError: 346671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9387.076391] Lustre: server umount lustre-MDT0001 complete [ 9399.157127] Lustre: server umount lustre-OST0000 complete [ 9405.710612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9405.920736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9405.925282] Lustre: Skipped 1 previous similar message [ 9407.547806] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9411.059543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.967826] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9416.484526] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9419.140614] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9421.944207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9424.343990] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9427.350402] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9430.310859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9440.737584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9440.741771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9440.746852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9440.749821] Lustre: Skipped 2 previous similar messages [ 9442.004154] Lustre: server umount lustre-OST0000 complete [ 9443.691600] Lustre: server umount lustre-MDT0000 complete [ 9445.351489] LustreError: 354885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944642 with bad export cookie 20138436248602206 [ 9445.364685] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9445.537380] Lustre: server umount lustre-MDT0001 complete [ 9450.107178] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9451.295370] Key type lgssc unregistered [ 9451.443431] LNet: 356590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9451.446197] LNetError: 356590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9451.458411] LNet: Removed LNI 192.168.201.136@tcp [ 9451.795945] Key type .llcrypt unregistered [ 9451.797257] Key type ._llcrypt unregistered [ 9463.717222] Key type ._llcrypt registered [ 9463.719238] Key type .llcrypt registered [ 9463.774276] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9464.175989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9464.226104] alg: No test for adler32 (adler32-zlib) [ 9465.098837] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9465.191417] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9466.778445] Key type lgssc registered [ 9467.198486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9470.390651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9470.403072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.593058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9473.428321] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9477.496377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.736250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9479.625840] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9483.020355] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9485.646170] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9488.469738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9488.612973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9490.998264] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9494.194868] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9496.819157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9513.441758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9513.443900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9513.445940] Lustre: Skipped 1 previous similar message [ 9513.447207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9513.448021] Lustre: Skipped 2 previous similar messages [ 9514.647490] Lustre: server umount lustre-MDT0000 complete [ 9516.079252] LustreError: 358109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944712 with bad export cookie 537319523351199397 [ 9516.082564] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9516.083828] LustreError: 358109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9516.209619] Lustre: server umount lustre-MDT0001 complete [ 9527.974370] Lustre: server umount lustre-OST0000 complete [ 9533.577264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9533.828530] LustreError: 361811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9533.837177] LustreError: 361811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9533.872719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9535.542365] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9539.040938] LustreError: 361812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9539.247185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9539.386887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9540.933639] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9544.242649] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9546.745447] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9549.868173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9550.021442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9552.553570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9555.585216] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9558.247150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9569.761353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9569.762960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9569.766550] Lustre: Skipped 3 previous similar messages [ 9569.769983] Lustre: Skipped 1 previous similar message [ 9574.057745] Lustre: server umount lustre-MDT0000 complete [ 9574.881778] LustreError: 361806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9574.891103] LustreError: 361806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9575.564197] LustreError: 361793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944772 with bad export cookie 537319523351200496 [ 9575.566528] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9575.568606] LustreError: 361793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9575.741887] Lustre: server umount lustre-MDT0001 complete [ 9587.670783] Lustre: server umount lustre-OST0000 complete [ 9593.738571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9593.929669] LustreError: 365718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9593.956936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9595.509207] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9598.805287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9600.490540] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9603.281530] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9605.463585] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9608.013610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9610.124151] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9613.087270] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9618.410882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9623.521473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9623.522227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9623.527526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9623.531404] Lustre: Skipped 1 previous similar message [ 9625.687698] Lustre: server umount lustre-OST0000 complete [ 9628.640799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9628.642648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9628.645455] Lustre: Skipped 2 previous similar messages [ 9628.651017] Lustre: Skipped 1 previous similar message [ 9633.201512] Lustre: server umount lustre-MDT0000 complete [ 9633.760070] LustreError: 365718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9633.765815] LustreError: 365718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9634.786641] LustreError: 365700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944831 with bad export cookie 537319523351201504 [ 9634.788285] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9634.790677] LustreError: 365700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9634.929287] Lustre: server umount lustre-MDT0001 complete [ 9639.669561] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9640.831516] Key type lgssc unregistered [ 9641.001759] LNet: 369247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9641.007120] LNetError: 369247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9641.017823] LNet: Removed LNI 192.168.201.136@tcp [ 9641.418207] Key type .llcrypt unregistered [ 9641.420580] Key type ._llcrypt unregistered [ 9651.811825] Key type ._llcrypt registered [ 9651.812920] Key type .llcrypt registered [ 9651.861896] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9652.209322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9652.240281] alg: No test for adler32 (adler32-zlib) [ 9653.100391] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9653.213383] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9654.815148] Key type lgssc registered [ 9655.253151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9658.203399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9658.213773] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9658.218743] LustreError: 370665:0:(osd_handler.c:8555:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9658.229569] LustreError: 370665:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9658.232420] LustreError: 370665:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9658.235384] LustreError: 370665:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9658.239074] LustreError: 370665:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9661.226066] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9661.231261] LustreError: 371062:0:(osd_handler.c:8555:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9661.240830] LustreError: 371062:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9661.244235] LustreError: 371062:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9661.248464] LustreError: 371062:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9661.251856] LustreError: 371062:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9669.094829] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9670.210301] Key type lgssc unregistered [ 9670.359852] LNet: 372075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9670.365821] LNetError: 372075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9670.378374] LNet: Removed LNI 192.168.201.136@tcp [ 9670.741355] Key type .llcrypt unregistered [ 9670.743093] Key type ._llcrypt unregistered [ 9677.697949] Key type ._llcrypt registered [ 9677.699100] Key type .llcrypt registered [ 9677.738430] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9684.605164] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9685.009428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9685.025548] alg: No test for adler32 (adler32-zlib) [ 9685.906684] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9685.997690] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9687.575186] Key type lgssc registered [ 9687.993892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9700.229669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9700.233852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9701.331197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9701.348801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9701.390480] Lustre: lustre-MDT0000: new disk, initializing [ 9701.413302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9701.419423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9702.950018] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9707.633501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.683049] Lustre: 375207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9707.704644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9707.706322] Lustre: Skipped 1 previous similar message [ 9707.741826] Lustre: lustre-MDT0001: new disk, initializing [ 9707.785285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9707.800391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9707.803870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9709.404883] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9713.809460] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9715.877701] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9718.454280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9718.582958] Lustre: lustre-OST0000: new disk, initializing [ 9718.587588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9718.590443] Lustre: 377173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9718.635463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9720.849523] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9721.657377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9721.661199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9721.695516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9724.929821] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9726.417112] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9726.514424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9727.981083] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9728.058269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9732.064980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9732.069125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.069967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9732.074871] Lustre: Skipped 1 previous similar message [ 9737.187460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9737.189892] Lustre: Skipped 2 previous similar messages [ 9742.305848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9742.308404] Lustre: Skipped 1 previous similar message [ 9742.815126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9742.889451] Lustre: server umount lustre-OST0000 complete [ 9747.424577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.425367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9747.428417] Lustre: Skipped 1 previous similar message [ 9750.298316] Lustre: server umount lustre-MDT0000 complete [ 9751.648490] LustreError: 375199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944948 with bad export cookie 11652794019485931122 [ 9751.651763] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9751.652369] LustreError: 375199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9751.779329] Lustre: server umount lustre-MDT0001 complete [ 9754.878198] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9755.462076] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:02:31 (1784944951) [ 9757.807995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9757.993533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9759.428533] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9762.701563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9762.839942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9764.370985] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9767.205644] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9769.310467] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9771.842088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9771.961885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9774.018570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9776.431224] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9778.691785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9778.756268] Lustre: lustre-OST0001: new disk, initializing [ 9778.758194] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9778.761459] Lustre: 382463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9780.839121] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9783.297247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9783.304990] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9783.334323] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9784.787506] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9788.385449] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9788.386359] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.388112] LustreError: Skipped 2 previous similar messages [ 9788.396523] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9788.400038] Lustre: Skipped 2 previous similar messages [ 9792.894599] Lustre: server umount lustre-OST0001 complete [ 9794.534407] LustreError: 381449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9794.539048] LustreError: 381449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9799.649463] LustreError: 383030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.654722] LustreError: 383030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9804.771214] LustreError: 381451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9804.776960] LustreError: 381451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9808.864424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9808.867173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.870890] Lustre: Skipped 1 previous similar message [ 9808.872812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9808.874543] Lustre: Skipped 3 previous similar messages [ 9809.891967] LustreError: 381449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.898155] LustreError: 381449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9811.404692] Lustre: server umount lustre-OST0000 complete [ 9813.984741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9813.988206] LustreError: Skipped 1 previous similar message [ 9813.990387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9813.995981] Lustre: Skipped 1 previous similar message [ 9825.248129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9825.249779] Lustre: Skipped 8 previous similar messages [ 9827.295152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9827.355254] Lustre: server umount lustre-MDT0000 complete [ 9828.612099] LustreError: 379656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945025 with bad export cookie 11652794019485932011 [ 9828.614896] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9828.616099] LustreError: 379656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9834.804142] Lustre: server umount lustre-MDT0001 complete [ 9838.407199] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9839.412241] Key type lgssc unregistered [ 9839.535356] LNet: 384448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9839.538254] LNetError: 384448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9839.548314] LNet: Removed LNI 192.168.201.136@tcp [ 9839.846311] Key type .llcrypt unregistered [ 9839.847742] Key type ._llcrypt unregistered [ 9859.191903] Key type ._llcrypt registered [ 9859.193000] Key type .llcrypt registered [ 9859.242205] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9859.699812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9859.711226] alg: No test for adler32 (adler32-zlib) [ 9860.584092] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [ 9860.680886] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9862.271201] Key type lgssc registered [ 9862.709572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9865.807842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9865.813846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.905801] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9866.913724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9866.988386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.473233] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9872.390364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9872.440772] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9872.453049] Lustre: 387220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9872.558601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9874.185650] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9877.084271] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9879.496748] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9880.544194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9880.544851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9880.551077] Lustre: Skipped 1 previous similar message [ 9880.557131] Lustre: Skipped 1 previous similar message [ 9885.663967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9885.666227] Lustre: Skipped 1 previous similar message [ 9886.424932] Lustre: server umount lustre-MDT0000 complete [ 9887.851155] LustreError: 387213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945084 with bad export cookie 15803438157414496609 [ 9887.853432] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9887.856445] LustreError: 387213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9887.980516] Lustre: server umount lustre-MDT0001 complete [ 9891.244920] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:04:47 (1784945087) [ 9893.640154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9893.844455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9895.384922] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9898.525542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.646574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9900.115716] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9902.701378] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9904.659535] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9907.147781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9907.216615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9907.295540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9909.550124] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9912.373577] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9916.301868] Lustre: Permanently deactivating lustre-OST0000 [ 9916.308838] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9926.645632] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9927.607578] LustreError: 392904:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9927.610880] LustreError: 392904:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 9927.613346] LustreError: 392904:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9928.272449] LustreError: 392953:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 9928.275585] LustreError: 392953:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9935.056288] Lustre: server umount lustre-OST0000 complete [ 9935.947762] LustreError: 393203:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 9935.950431] LustreError: 393203:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9939.424637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9939.425149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9939.428436] Lustre: Skipped 1 previous similar message [ 9944.544351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9944.546579] Lustre: Skipped 2 previous similar messages [ 9946.622315] Lustre: server umount lustre-MDT0000 complete [ 9948.019319] LustreError: 389709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945144 with bad export cookie 15803438157414497176 [ 9948.022706] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9948.023096] LustreError: 389709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9957.415636] Lustre: server umount lustre-MDT0001 complete [ 9960.407562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9961.924725] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9962.894608] LustreError: 394361:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9965.301551] Lustre: 394699:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.136@tcp [ 9966.087305] Lustre: server umount MGS complete [ 9970.341293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.510114] LustreError: 395423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.518037] LustreError: 395423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9970.518857] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9970.523305] Lustre: Skipped 1 previous similar message [ 9970.548663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9972.147412] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9975.211739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.354720] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9977.195723] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9980.162299] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9982.641966] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9983.510597] Lustre: Permanently reactivating lustre-OST0000 [ 9983.516127] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9983.518853] Lustre: Skipped 2 previous similar messages [10007.519093] LustreError: 397151:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10009.995940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10010.103145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10010.105283] Lustre: Skipped 1 previous similar message [10011.984390] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10014.560428] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10027.488573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10027.491548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10027.492459] Lustre: Skipped 1 previous similar message [10027.495619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10028.370748] Lustre: server umount lustre-OST0000 complete [10031.583795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10031.585833] LustreError: Skipped 1 previous similar message [10031.587091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10031.771743] Lustre: server umount lustre-MDT0000 complete [10032.608504] LustreError: 395423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10032.613753] LustreError: 395423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10033.035628] LustreError: 395405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945229 with bad export cookie 15803438157414498212 [10033.039131] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10033.039941] LustreError: 395405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10033.161804] Lustre: server umount lustre-MDT0001 complete [10036.739423] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10037.823623] Key type lgssc unregistered [10037.958547] LNet: 399998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10037.962721] LNetError: 399998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10037.974322] LNet: Removed LNI 192.168.201.136@tcp [10038.291511] Key type .llcrypt unregistered [10038.293095] Key type ._llcrypt unregistered [10044.425682] Key type ._llcrypt registered [10044.427198] Key type .llcrypt registered [10044.467242] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [10050.766619] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10051.190622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10051.238494] alg: No test for adler32 (adler32-zlib) [10052.086735] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [10052.179801] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10053.759181] Key type lgssc registered [10054.138286] Lustre: Echo OBD driver; http://www.lustre.org/ [10065.639288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10065.644574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10066.739866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10066.752026] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10066.788179] Lustre: lustre-MDT0000: new disk, initializing [10066.810472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10066.817807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10068.262755] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10072.626053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10072.680962] Lustre: 403128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10072.713747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10072.715984] Lustre: Skipped 1 previous similar message [10072.753834] Lustre: lustre-MDT0001: new disk, initializing [10072.793316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10072.815549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10072.819865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10074.315217] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10078.415410] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10080.601610] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10083.063383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10083.180654] Lustre: lustre-OST0000: new disk, initializing [10083.182692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10083.185817] Lustre: 405095:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10083.220518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10085.378128] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10088.954748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10088.958392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10089.006059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10089.317871] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10090.780681] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10090.853965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10092.217261] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10092.289726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10094.050621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10094.050777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10094.055914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10094.060762] Lustre: Skipped 1 previous similar message [10099.099784] Lustre: server umount lustre-OST0000 complete [10103.775967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10103.778644] LustreError: Skipped 1 previous similar message [10103.780598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10103.784707] Lustre: Skipped 1 previous similar message [10103.786875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10106.524546] Lustre: server umount lustre-MDT0000 complete [10107.736248] LustreError: 403165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945304 with bad export cookie 11012561593696511191 [10107.738615] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10107.740293] LustreError: 403165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10107.864755] Lustre: server umount lustre-MDT0001 complete [10110.610021] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10111.163959] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:08:27 (1784945307) [10113.333792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10113.489990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10114.822228] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10117.669233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10117.795673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10119.093244] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10121.515643] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10123.298418] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10125.505258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10127.489274] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10129.937884] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10130.642350] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10141.752437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10145.250735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10145.251505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10145.251530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10145.254042] Lustre: Skipped 2 previous similar messages [10145.255728] Lustre: Skipped 2 previous similar messages [10149.389706] Lustre: server umount lustre-OST0000 complete [10153.951792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10153.954191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10153.959249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10153.961236] Lustre: Skipped 1 previous similar message [10156.751861] Lustre: server umount lustre-MDT0000 complete [10158.026419] LustreError: 407530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945354 with bad export cookie 11012561593696512059 [10158.029404] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10158.032931] LustreError: 407530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10158.152839] Lustre: server umount lustre-MDT0001 complete [10162.070135] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10163.237460] Key type lgssc unregistered [10163.363510] LNet: 411292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10163.366268] LNetError: 411292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10163.376341] LNet: Removed LNI 192.168.201.136@tcp [10163.727113] Key type .llcrypt unregistered [10163.728359] Key type ._llcrypt unregistered [10172.368897] Key type ._llcrypt registered [10172.370255] Key type .llcrypt registered [10172.414346] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10172.521219] Key type .llcrypt unregistered [10172.522226] Key type ._llcrypt unregistered [10180.334169] Key type ._llcrypt registered [10180.335612] Key type .llcrypt registered [10180.386169] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10180.746155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10180.791479] alg: No test for adler32 (adler32-zlib) [10181.654801] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [10181.742106] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10183.327128] Key type lgssc registered [10183.727812] Lustre: Echo OBD driver; http://www.lustre.org/ [10186.597897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10186.604711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10187.777916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10189.124677] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10192.054879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10192.203754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10194.211799] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10196.570108] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10197.345359] LustreError: 413479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10197.351112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10197.351549] LustreError: 413479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10199.025473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10199.197791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10200.608923] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10209.764043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10209.765024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10209.769314] Lustre: Skipped 1 previous similar message [10209.773240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10213.994680] Lustre: server umount lustre-OST0000 complete [10218.978382] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10218.981584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10218.986288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10218.988244] Lustre: Skipped 1 previous similar message [10219.999936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10220.000540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.460127] Lustre: server umount lustre-MDT0000 complete [10222.779252] LustreError: 413459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945419 with bad export cookie 2573450541865787231 [10222.782438] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10222.783182] LustreError: 413459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10222.902440] Lustre: server umount lustre-MDT0001 complete [10226.667313] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10227.692765] Key type lgssc unregistered [10227.835471] LNet: 416622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10227.838332] LNetError: 416622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10227.846309] LNet: Removed LNI 192.168.201.136@tcp [10228.205720] Key type .llcrypt unregistered [10228.207108] Key type ._llcrypt unregistered [10237.904170] Key type ._llcrypt registered [10237.905222] Key type .llcrypt registered [10237.941043] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10238.406194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10238.418427] alg: No test for adler32 (adler32-zlib) [10239.292338] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [10239.391345] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10240.983138] Key type lgssc registered [10241.379088] Lustre: Echo OBD driver; http://www.lustre.org/ [10244.261630] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10244.269269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10259.679347] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10264.799330] LustreError: 417988:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.136@tcp: failed processing log, type 4: rc = -110 [10291.423140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10293.141810] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10295.392125] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10297.645148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10297.802207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10297.830871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10299.062184] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10301.579894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10301.700823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10303.042873] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10305.282801] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10307.023204] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10312.161305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10312.164108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10312.167757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10312.169833] Lustre: Skipped 1 previous similar message [10315.340725] Lustre: server umount lustre-OST0000 complete [10317.280707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10317.280893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.281560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10317.288684] Lustre: Skipped 2 previous similar messages [10322.399983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10322.401878] Lustre: Skipped 2 previous similar messages [10322.645968] Lustre: server umount lustre-MDT0000 complete [10323.775043] LustreError: 417996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945520 with bad export cookie 7207231220318534061 [10323.778218] LustreError: 417996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10323.778471] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10323.881283] Lustre: server umount lustre-MDT0001 complete [10327.162856] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10328.111372] Key type lgssc unregistered [10328.223312] LNet: 421556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10328.225437] LNetError: 421556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10328.232261] LNet: Removed LNI 192.168.201.136@tcp [10328.517492] Key type .llcrypt unregistered [10328.518382] Key type ._llcrypt unregistered [10337.225741] Key type ._llcrypt registered [10337.226655] Key type .llcrypt registered [10337.261500] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10337.600469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10337.625596] alg: No test for adler32 (adler32-zlib) [10338.472333] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [10338.562221] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10340.143149] Key type lgssc registered [10340.516361] Lustre: Echo OBD driver; http://www.lustre.org/ [10342.964384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10342.971290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.110130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10345.391429] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10348.075639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.198916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10349.458287] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10351.701647] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10353.335627] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10355.480902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10355.579732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10357.330847] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10359.644453] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10369.964034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10374.112772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10374.115280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10374.121985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10375.136516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10375.138096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10375.139164] Lustre: Skipped 1 previous similar message [10375.145689] Lustre: Skipped 1 previous similar message [10377.104896] Lustre: server umount lustre-MDT0000 complete [10377.849602] LustreError: 424802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.36@tcp arrived at 1784945574 with bad export cookie 17477309999708434433 [10377.853371] LustreError: 424802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10378.472095] LustreError: 422941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945575 with bad export cookie 17477309999708433782 [10378.475360] LustreError: 422941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10378.475773] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10379.231787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10379.234152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10379.238626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10379.240337] Lustre: Skipped 2 previous similar messages [10380.255829] LustreError: 422959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10380.259685] LustreError: 422959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10383.263988] LustreError: 422960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10383.264456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10383.272033] LustreError: 422960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10383.285039] Lustre: Skipped 1 previous similar message [10384.586112] Lustre: server umount lustre-OST0000 complete [10386.722458] Lustre: server umount lustre-MDT0001 complete [10389.812329] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10390.715953] Key type lgssc unregistered [10390.832452] LNet: 426664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10390.834598] LNetError: 426664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10390.846270] LNet: Removed LNI 192.168.201.136@tcp [10391.127073] Key type .llcrypt unregistered [10391.128703] Key type ._llcrypt unregistered [10399.471527] Key type ._llcrypt registered [10399.472402] Key type .llcrypt registered [10399.503503] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10399.790896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10399.821076] alg: No test for adler32 (adler32-zlib) [10400.681878] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [10400.774968] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10402.351111] Key type lgssc registered [10402.655300] Lustre: Echo OBD driver; http://www.lustre.org/ [10404.949374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10404.954326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.075595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10407.234639] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10409.778079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10409.882122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10411.040433] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10413.244683] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10414.895030] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10416.951881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10417.060742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10419.040369] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10421.239324] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10431.398851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10435.551924] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10435.554609] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10435.559881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10436.577429] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10436.580412] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10441.696698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10441.698105] Lustre: Skipped 2 previous similar messages [10446.817275] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10446.819373] Lustre: Skipped 2 previous similar messages [10447.327158] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10447.384268] Lustre: server umount lustre-MDT0001 complete [10451.936510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10451.937059] LustreError: 428056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.938895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10451.946538] LustreError: 428056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10451.950824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10451.954536] Lustre: Skipped 1 previous similar message [10455.118225] Lustre: server umount lustre-OST0000 complete [10456.450924] Lustre: server umount lustre-MDT0000 complete [10461.163316] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10462.236313] Key type lgssc unregistered [10462.378465] LNet: 431759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10462.382642] LNetError: 431759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10462.394440] LNet: Removed LNI 192.168.201.136@tcp [10462.753853] Key type .llcrypt unregistered [10462.754931] Key type ._llcrypt unregistered [10471.043093] Key type ._llcrypt registered [10471.044123] Key type .llcrypt registered [10471.082076] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [10477.207558] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10477.629187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10477.642464] alg: No test for adler32 (adler32-zlib) [10478.502165] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [10478.591956] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10480.175652] Key type lgssc registered [10480.578175] Lustre: Echo OBD driver; http://www.lustre.org/ [10492.934863] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10497.556279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10497.560284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10498.632840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10498.641659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10498.674395] Lustre: lustre-MDT0000: new disk, initializing [10498.692108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10498.697055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10499.859930] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10505.027579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10505.114814] Lustre: lustre-OST0000: new disk, initializing [10505.116274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10505.117611] Lustre: Skipped 1 previous similar message [10505.119323] Lustre: 436761:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10505.139118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10505.782163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10505.786744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10505.795012] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10506.796100] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10510.427670] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10515.798378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.843091] Lustre: 435746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10515.871109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10515.909680] Lustre: lustre-MDT0001: new disk, initializing [10515.956340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10515.971364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10515.975571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10517.386432] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10520.978567] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10521.060417] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10522.382719] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10526.523413] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 4 sec [10529.271318] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [10530.785894] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10530.786028] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.788799] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10530.792795] Lustre: Skipped 1 previous similar message [10530.795919] Lustre: Skipped 1 previous similar message [10534.289265] Lustre: server umount lustre-MDT0001 complete [10541.025672] LustreError: 438450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10541.033468] LustreError: 438450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10546.146767] LustreError: 438450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10546.151071] LustreError: 438450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10546.655977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10546.659976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10546.666256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10551.264838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10551.264965] LustreError: 435755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.661644] Lustre: server umount lustre-OST0000 complete [10560.204814] Lustre: server umount lustre-MDT0000 complete [10564.848361] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10565.790310] Key type lgssc unregistered [10565.923522] LNet: 440535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10565.926160] LNetError: 440535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10565.935397] LNet: Removed LNI 192.168.201.136@tcp [10566.244084] Key type .llcrypt unregistered [10566.244933] Key type ._llcrypt unregistered [10578.301378] Key type ._llcrypt registered [10578.302513] Key type .llcrypt registered [10578.345238] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10578.457357] Key type .llcrypt unregistered [10578.458473] Key type ._llcrypt unregistered [10584.991766] Key type ._llcrypt registered [10584.992953] Key type .llcrypt registered [10585.030192] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10585.422921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10585.459680] alg: No test for adler32 (adler32-zlib) [10586.322738] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [10586.408156] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10587.983159] Key type lgssc registered [10588.330975] Lustre: Echo OBD driver; http://www.lustre.org/ [10597.086438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10597.095413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10598.177993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10598.189271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10598.234095] Lustre: lustre-MDT0000: new disk, initializing [10598.256913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10598.264583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10599.673420] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10603.780491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10603.826281] Lustre: 443882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10603.852218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10603.854892] Lustre: Skipped 1 previous similar message [10603.890236] Lustre: lustre-MDT0001: new disk, initializing [10603.922617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10603.938661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10603.942637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10605.320739] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10609.061240] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10610.953327] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10613.257145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10613.378575] Lustre: lustre-OST0000: new disk, initializing [10613.380403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10613.384265] Lustre: 445850:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10613.412539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10615.377707] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10619.223358] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10621.950640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10621.955759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10621.994239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10627.041198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10627.043063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10627.044587] Lustre: Skipped 2 previous similar messages [10627.046441] Lustre: Skipped 1 previous similar message [10632.163616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10632.165794] Lustre: Skipped 3 previous similar messages [10632.792222] Lustre: server umount lustre-MDT0000 complete [10634.041386] LustreError: 444677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945830 with bad export cookie 15557183895566735256 [10634.044840] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10634.045726] LustreError: 444677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10634.152161] Lustre: server umount lustre-MDT0001 complete [10645.652602] Lustre: server umount lustre-OST0000 complete [10648.759884] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:17:24 (1784945844) [10651.080063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10651.230282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10652.469665] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10655.155599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10655.276561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10656.541448] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10658.760286] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10660.437530] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10675.490310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10675.558736] Lustre: Found index 0 for lustre-OST0000, updating log [10675.622393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10676.648025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10677.568887] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10680.003325] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10681.823968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10681.826616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.831298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10685.923350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10685.927071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10685.935246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10687.249659] Lustre: server umount lustre-OST0000 complete [10701.279669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10701.283179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10701.289047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10701.290697] Lustre: Skipped 2 previous similar messages [10713.568125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10713.569787] Lustre: Skipped 5 previous similar messages [10714.079069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10714.140360] Lustre: server umount lustre-MDT0000 complete [10714.592372] LustreError: 447897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10714.599022] LustreError: 447897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10715.422412] LustreError: 447860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945912 with bad export cookie 15557183895566736649 [10715.425025] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10715.426971] LustreError: 447860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10715.557786] Lustre: server umount lustre-MDT0001 complete [10718.249422] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:18:34 (1784945914) [10735.903299] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp [11297.846036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11298.012206] LustreError: 454278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11298.040462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11299.360536] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11302.068518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11303.483154] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11305.764770] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11307.530321] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11322.725307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11322.835065] Lustre: Found index 0 for lustre-OST0000, updating log [11324.781983] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11324.968236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11327.336499] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11329.063308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11329.072381] Lustre: Skipped 1 previous similar message [11329.076586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11334.161699] Lustre: server umount lustre-OST0000 complete [11335.648730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11335.652969] Lustre: Skipped 1 previous similar message [11341.532043] Lustre: server umount lustre-MDT0000 complete [11341.792852] LustreError: 456097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11341.797328] LustreError: 456097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11342.871466] LustreError: 454259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946539 with bad export cookie 15557183895566737713 [11342.875265] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11342.875833] LustreError: 454259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11342.994693] Lustre: server umount lustre-MDT0001 complete [11349.279193] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp [11360.064570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [11363.157557] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11374.219572] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:29:30 (1784946570) [11379.852811] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11380.815186] Key type lgssc unregistered [11385.055428] LNet: 460933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11385.057922] LNetError: 460933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11453.669509] LNet: Removed LNI 192.168.201.136@tcp [11454.021911] Key type .llcrypt unregistered [11454.023151] Key type ._llcrypt unregistered [11460.427355] Key type ._llcrypt registered [11460.428399] Key type .llcrypt registered [11460.468107] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11460.879863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11460.886739] alg: No test for adler32 (adler32-zlib) [11461.731105] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [11461.815606] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11463.391110] Key type lgssc registered [11463.759436] Lustre: Echo OBD driver; http://www.lustre.org/ [11465.137473] LNet: Added LNI 192.168.201.136@tcp100 [8/256/0/180] [11465.429301] LNet: Added LNI 192.168.201.136@tcp101 [8/256/0/180] [11465.710646] LNet: Added LNI 192.168.201.136@tcp102 [8/256/0/180] [11465.996470] LNet: Added LNI 192.168.201.136@tcp103 [8/256/0/180] [11466.302162] LNet: Added LNI 192.168.201.136@tcp104 [8/256/0/180] [11466.591154] LNet: Added LNI 192.168.201.136@tcp105 [8/256/0/180] [11466.870171] LNet: Added LNI 192.168.201.136@tcp106 [8/256/0/180] [11467.142668] LNet: Added LNI 192.168.201.136@tcp107 [8/256/0/180] [11467.436161] LNet: Added LNI 192.168.201.136@tcp108 [8/256/0/180] [11467.722899] LNet: Added LNI 192.168.201.136@tcp109 [8/256/0/180] [11468.028262] LNet: Added LNI 192.168.201.136@tcp110 [8/256/0/180] [11468.328533] LNet: Added LNI 192.168.201.136@tcp111 [8/256/0/180] [11468.628969] LNet: Added LNI 192.168.201.136@tcp112 [8/256/0/180] [11468.920598] LNet: Added LNI 192.168.201.136@tcp113 [8/256/0/180] [11469.208283] LNet: Added LNI 192.168.201.136@tcp114 [8/256/0/180] [11469.474211] LNet: Added LNI 192.168.201.136@tcp115 [8/256/0/180] [11469.748097] LNet: Added LNI 192.168.201.136@tcp116 [8/256/0/180] [11470.033510] LNet: Added LNI 192.168.201.136@tcp117 [8/256/0/180] [11470.312548] LNet: Added LNI 192.168.201.136@tcp118 [8/256/0/180] [11470.593230] LNet: Added LNI 192.168.201.136@tcp119 [8/256/0/180] [11470.872862] LNet: Added LNI 192.168.201.136@tcp120 [8/256/0/180] [11471.149079] LNet: Added LNI 192.168.201.136@tcp121 [8/256/0/180] [11471.430220] LNet: Added LNI 192.168.201.136@tcp122 [8/256/0/180] [11471.711724] LNet: Added LNI 192.168.201.136@tcp123 [8/256/0/180] [11472.008359] LNet: Added LNI 192.168.201.136@tcp124 [8/256/0/180] [11472.298146] LNet: Added LNI 192.168.201.136@tcp125 [8/256/0/180] [11472.585446] LNet: Added LNI 192.168.201.136@tcp126 [8/256/0/180] [11472.865799] LNet: Added LNI 192.168.201.136@tcp127 [8/256/0/180] [11473.157210] LNet: Added LNI 192.168.201.136@tcp128 [8/256/0/180] [11473.443592] LNet: Added LNI 192.168.201.136@tcp129 [8/256/0/180] [11473.720249] LNet: Added LNI 192.168.201.136@tcp130 [8/256/0/180] [11474.007889] LNet: Added LNI 192.168.201.136@tcp131 [8/256/0/180] [11474.296290] LNet: Added LNI 192.168.201.136@tcp132 [8/256/0/180] [11474.588971] LNet: Added LNI 192.168.201.136@tcp133 [8/256/0/180] [11474.894269] LNet: Added LNI 192.168.201.136@tcp134 [8/256/0/180] [11475.203166] LNet: Added LNI 192.168.201.136@tcp135 [8/256/0/180] [11477.316700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11477.321112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11478.357800] Lustre: 463929:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [11478.416127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11478.425417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11478.461289] Lustre: lustre-MDT0000: new disk, initializing [11478.484574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11478.490439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11479.862982] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11484.068613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11484.127482] Lustre: 463944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11484.164365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11484.166542] Lustre: Skipped 1 previous similar message [11484.206619] Lustre: lustre-MDT0001: new disk, initializing [11484.218772] Lustre: 464744:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [11484.222539] Lustre: 464744:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11484.269668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11484.284888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11484.288731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11485.749055] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11487.606467] Lustre: 465131:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [11487.612398] Lustre: 465131:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11489.408125] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11491.347724] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11493.759182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11493.876334] Lustre: lustre-OST0000: new disk, initializing [11493.878148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11493.882468] Lustre: 465919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11493.917364] Lustre: 465924:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [11493.919886] Lustre: 465924:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11493.937200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11495.551409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11495.555688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11495.603611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11495.739956] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11499.704286] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11506.145659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11506.146215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11506.147769] LustreError: Skipped 1 previous similar message [11506.151635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11506.154114] Lustre: Skipped 1 previous similar message [11508.431303] Lustre: server umount lustre-OST0000 complete [11509.727878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11509.731943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11509.735818] Lustre: Skipped 1 previous similar message [11509.737745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11511.264524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11511.265702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11515.807687] Lustre: server umount lustre-MDT0000 complete [11516.964175] LustreError: 463938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946713 with bad export cookie 17361613042892404419 [11516.967898] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11516.968304] LustreError: 463938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11517.100091] Lustre: server umount lustre-MDT0001 complete [11520.666205] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11521.631516] Key type lgssc unregistered [11521.774684] LNet: 467831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11521.777377] LNetError: 467831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11521.779257] LNet: Removed LNI 192.168.201.136@tcp [11522.783146] LNet: Removed LNI 192.168.201.136@tcp100 [11522.784283] LNetError: 467831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11522.786778] LNetError: 467831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11522.790104] LNet: Removed LNI 192.168.201.136@tcp101 [11523.807100] LNet: Removed LNI 192.168.201.136@tcp102 [11523.808273] LNetError: 467831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11523.810158] LNetError: 467831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11523.812015] LNet: Removed LNI 192.168.201.136@tcp103 [11524.831117] LNet: Removed LNI 192.168.201.136@tcp104 [11525.855178] LNet: Removed LNI 192.168.201.136@tcp105 [11525.857832] LNetError: 467831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11525.862221] LNetError: 467831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11525.867257] LNet: Removed LNI 192.168.201.136@tcp106 [11526.879112] LNet: Removed LNI 192.168.201.136@tcp107 [11527.903144] LNet: Removed LNI 192.168.201.136@tcp108 [11528.927137] LNet: Removed LNI 192.168.201.136@tcp109 [11529.951139] LNet: Removed LNI 192.168.201.136@tcp110 [11529.953264] LNetError: 467831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11529.956973] LNetError: 467831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11529.960797] LNet: Removed LNI 192.168.201.136@tcp111 [11529.962684] LNet: Removed LNI 192.168.201.136@tcp112 [11529.964668] LNet: Removed LNI 192.168.201.136@tcp113 [11529.966656] LNet: Removed LNI 192.168.201.136@tcp114 [11529.968545] LNet: Removed LNI 192.168.201.136@tcp115 [11529.970683] LNet: Removed LNI 192.168.201.136@tcp116 [11529.972663] LNet: Removed LNI 192.168.201.136@tcp117 [11529.974732] LNet: Removed LNI 192.168.201.136@tcp118 [11529.976608] LNet: Removed LNI 192.168.201.136@tcp119 [11529.978661] LNet: Removed LNI 192.168.201.136@tcp120 [11529.980694] LNet: Removed LNI 192.168.201.136@tcp121 [11529.982641] LNet: Removed LNI 192.168.201.136@tcp122 [11529.984608] LNet: Removed LNI 192.168.201.136@tcp123 [11529.986572] LNet: Removed LNI 192.168.201.136@tcp124 [11529.988551] LNet: Removed LNI 192.168.201.136@tcp125 [11529.990522] LNet: Removed LNI 192.168.201.136@tcp126 [11529.992453] LNet: Removed LNI 192.168.201.136@tcp127 [11529.994546] LNet: Removed LNI 192.168.201.136@tcp128 [11529.996767] LNet: Removed LNI 192.168.201.136@tcp129 [11529.998839] LNet: Removed LNI 192.168.201.136@tcp130 [11530.000894] LNet: Removed LNI 192.168.201.136@tcp131 [11530.002930] LNet: Removed LNI 192.168.201.136@tcp132 [11530.004985] LNet: Removed LNI 192.168.201.136@tcp133 [11530.007055] LNet: Removed LNI 192.168.201.136@tcp134 [11530.015641] LNet: Removed LNI 192.168.201.136@tcp135 [11530.387093] Key type .llcrypt unregistered [11530.388379] Key type ._llcrypt unregistered [11537.774974] Key type ._llcrypt registered [11537.775839] Key type .llcrypt registered [11537.809575] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [11544.051348] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11544.379415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11544.429863] alg: No test for adler32 (adler32-zlib) [11545.286502] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [11545.372313] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11546.943133] Key type lgssc registered [11547.298773] Lustre: Echo OBD driver; http://www.lustre.org/ [11559.407295] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:32:35 (1784946755) [11566.163732] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11567.239989] Key type lgssc unregistered [11567.360403] LNet: 471886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11567.363199] LNetError: 471886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11567.373337] LNet: Removed LNI 192.168.201.136@tcp [11567.687386] Key type .llcrypt unregistered [11567.689077] Key type ._llcrypt unregistered [11574.915367] Key type ._llcrypt registered [11574.916237] Key type .llcrypt registered [11574.953103] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11575.266487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11575.296882] alg: No test for adler32 (adler32-zlib) [11576.148406] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [11576.241708] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11577.823160] Key type lgssc registered [11578.171544] Lustre: Echo OBD driver; http://www.lustre.org/ [11581.211406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11581.216346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11582.300559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11582.308771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11582.345050] Lustre: lustre-MDT0000: new disk, initializing [11582.366386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11582.371912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11583.785786] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11587.784509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11587.827150] Lustre: 473074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11587.853590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11587.855890] Lustre: Skipped 1 previous similar message [11587.890479] Lustre: lustre-MDT0001: new disk, initializing [11587.919898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11587.931120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11587.933909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11589.179222] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11592.850200] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11594.615755] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11597.431539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11597.535969] Lustre: lustre-OST0000: new disk, initializing [11597.537401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11597.539445] Lustre: 475040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11597.567102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11599.798086] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11600.707473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11600.715207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11600.762941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11604.667954] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11605.972627] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing lnet_if_list [11607.551413] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11607.565088] LNet: Added LNI 192.168.201.136@tcp10 [8/256/0/180] [11609.649499] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11609.668094] LNet: Added LNI 192.168.201.136@tcp11 [8/256/0/180] [11611.242825] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11611.252292] LNet: Added LNI 192.168.201.136@tcp12 [8/256/0/180] [11612.537838] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11612.547404] LNet: Added LNI 192.168.201.136@tcp13 [8/256/0/180] [11614.083981] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11614.095099] LNet: Added LNI 192.168.201.136@tcp14 [8/256/0/180] [11615.757606] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11615.766696] LNet: Added LNI 192.168.201.136@tcp15 [8/256/0/180] [11617.387686] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11617.400613] LNet: Added LNI 192.168.201.136@tcp16 [8/256/0/180] [11619.129549] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11619.141747] LNet: Added LNI 192.168.201.136@tcp17 [8/256/0/180] [11620.689048] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11620.705762] LNet: Added LNI 192.168.201.136@tcp18 [8/256/0/180] [11622.609064] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11622.623446] LNet: Added LNI 192.168.201.136@tcp19 [8/256/0/180] [11624.611361] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11624.623657] LNet: Added LNI 192.168.201.136@tcp20 [8/256/0/180] [11626.874094] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11626.893617] LNet: Added LNI 192.168.201.136@tcp21 [8/256/0/180] [11628.950414] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11628.965432] LNet: Added LNI 192.168.201.136@tcp22 [8/256/0/180] [11631.110143] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11631.124479] LNet: Added LNI 192.168.201.136@tcp23 [8/256/0/180] [11633.145798] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11633.160674] LNet: Added LNI 192.168.201.136@tcp24 [8/256/0/180] [11635.163867] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11635.174841] LNet: Added LNI 192.168.201.136@tcp25 [8/256/0/180] [11637.262704] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11637.273706] LNet: Added LNI 192.168.201.136@tcp26 [8/256/0/180] [11639.195990] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11639.208557] LNet: Added LNI 192.168.201.136@tcp27 [8/256/0/180] [11641.079627] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11641.089862] LNet: Added LNI 192.168.201.136@tcp28 [8/256/0/180] [11642.980989] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11642.991581] LNet: Added LNI 192.168.201.136@tcp29 [8/256/0/180] [11644.829580] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11644.841318] LNet: Added LNI 192.168.201.136@tcp30 [8/256/0/180] [11646.786694] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11646.795800] LNet: Added LNI 192.168.201.136@tcp31 [8/256/0/180] [11648.758348] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11648.769867] LNet: Added LNI 192.168.201.136@tcp32 [8/256/0/180] [11650.814859] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11650.824844] LNet: Added LNI 192.168.201.136@tcp33 [8/256/0/180] [11652.864065] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11652.875245] LNet: Added LNI 192.168.201.136@tcp34 [8/256/0/180] [11655.184150] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11655.204204] LNet: Added LNI 192.168.201.136@tcp35 [8/256/0/180] [11657.369286] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11657.383262] LNet: Added LNI 192.168.201.136@tcp36 [8/256/0/180] [11659.661918] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11659.675725] LNet: Added LNI 192.168.201.136@tcp37 [8/256/0/180] [11662.042936] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11662.055181] LNet: Added LNI 192.168.201.136@tcp38 [8/256/0/180] [11664.304845] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11664.314786] LNet: Added LNI 192.168.201.136@tcp39 [8/256/0/180] [11666.742730] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11666.754385] LNet: Added LNI 192.168.201.136@tcp40 [8/256/0/180] [11669.069480] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11669.081792] LNet: Added LNI 192.168.201.136@tcp41 [8/256/0/180] [11671.634953] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11671.649572] LNet: Added LNI 192.168.201.136@tcp42 [8/256/0/180] [11674.194994] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11674.206309] LNet: Added LNI 192.168.201.136@tcp43 [8/256/0/180] [11676.867647] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11676.876828] LNet: Added LNI 192.168.201.136@tcp44 [8/256/0/180] [11679.270556] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11679.282367] LNet: Added LNI 192.168.201.136@tcp45 [8/256/0/180] [11681.970545] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11681.980817] LNet: Added LNI 192.168.201.136@tcp46 [8/256/0/180] [11685.056731] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11685.069523] LNet: Added LNI 192.168.201.136@tcp47 [8/256/0/180] [11688.202092] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11688.214890] LNet: Added LNI 192.168.201.136@tcp48 [8/256/0/180] [11691.057270] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11691.073360] LNet: Added LNI 192.168.201.136@tcp49 [8/256/0/180] [11693.964938] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11693.975529] LNet: Added LNI 192.168.201.136@tcp50 [8/256/0/180] [11703.265071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11703.267770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11703.271677] Lustre: Skipped 2 previous similar messages [11703.274321] Lustre: Skipped 2 previous similar messages [11708.130918] Lustre: server umount lustre-MDT0000 complete [11709.637901] LustreError: 473069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946906 with bad export cookie 14809814367216639677 [11709.643351] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11709.645215] LustreError: 473069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11709.775483] Lustre: server umount lustre-MDT0001 complete [11717.227587] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11720.133701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11729.887144] Lustre: 472349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784946910/real 1784946910] req@00000000e4abd136 x1871652347531776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784946926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11729.898351] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11730.911151] Lustre: 472349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784946911/real 1784946911] req@0000000002a246d8 x1871652347531904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784946927 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11735.007202] Lustre: 472350:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784946915/real 1784946915] req@00000000471c6f85 x1871652347532032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784946931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11735.007478] LustreError: 472347:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000164f4629 x1871652347532928/t0(0) o250->MGC192.168.201.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11735.108663] LustreError: 475034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11735.113230] LustreError: 475034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11735.121616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11736.433965] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11738.146058] LustreError: 475035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11739.168275] LustreError: 499229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11739.179465] LustreError: 499229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11739.654422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.775938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11741.070170] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11741.151163] Lustre: 472349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784946921/real 1784946921] req@00000000706412d8 x1871652347532416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784946937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11741.157352] Lustre: 472349:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11743.499480] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11745.253138] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11745.427542] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11746.271146] Lustre: 472348:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784946926/real 1784946926] req@0000000063fd2e75 x1871652347532544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784946942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11755.491080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11755.491244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11755.493836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11755.494413] LustreError: Skipped 1 previous similar message [11755.497643] Lustre: Skipped 1 previous similar message [11759.508982] Lustre: server umount lustre-OST0000 complete [11765.729256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11765.729844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11765.730513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11765.730518] Lustre: Skipped 1 previous similar message [11765.744611] Lustre: Skipped 1 previous similar message [11767.091827] Lustre: server umount lustre-MDT0000 complete [11768.711645] LustreError: 473068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946965 with bad export cookie 14809814367216641728 [11768.718452] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11768.719756] LustreError: 473068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11768.880568] Lustre: server umount lustre-MDT0001 complete [11772.587701] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11773.653077] Key type lgssc unregistered [11773.860522] LNet: 502006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11773.868487] LNetError: 502006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11773.875857] LNet: Removed LNI 192.168.201.136@tcp [11773.879422] LNet: Removed LNI 192.168.201.136@tcp10 [11773.882233] LNet: Removed LNI 192.168.201.136@tcp11 [11773.885730] LNet: Removed LNI 192.168.201.136@tcp12 [11773.890122] LNet: Removed LNI 192.168.201.136@tcp13 [11773.893688] LNet: Removed LNI 192.168.201.136@tcp14 [11773.896222] LNet: Removed LNI 192.168.201.136@tcp15 [11773.898682] LNet: Removed LNI 192.168.201.136@tcp16 [11773.901217] LNet: Removed LNI 192.168.201.136@tcp17 [11773.904992] LNet: Removed LNI 192.168.201.136@tcp18 [11773.907239] LNet: Removed LNI 192.168.201.136@tcp19 [11773.912050] LNet: Removed LNI 192.168.201.136@tcp20 [11773.915245] LNet: Removed LNI 192.168.201.136@tcp21 [11773.918915] LNet: Removed LNI 192.168.201.136@tcp22 [11773.922213] LNet: Removed LNI 192.168.201.136@tcp23 [11773.924445] LNet: Removed LNI 192.168.201.136@tcp24 [11773.927144] LNet: Removed LNI 192.168.201.136@tcp25 [11773.930240] LNet: Removed LNI 192.168.201.136@tcp26 [11773.932202] LNet: Removed LNI 192.168.201.136@tcp27 [11773.934115] LNet: Removed LNI 192.168.201.136@tcp28 [11773.936272] LNet: Removed LNI 192.168.201.136@tcp29 [11773.938139] LNet: Removed LNI 192.168.201.136@tcp30 [11773.940053] LNet: Removed LNI 192.168.201.136@tcp31 [11773.941909] LNet: Removed LNI 192.168.201.136@tcp32 [11773.944685] LNet: Removed LNI 192.168.201.136@tcp33 [11773.947328] LNet: Removed LNI 192.168.201.136@tcp34 [11773.949464] LNet: Removed LNI 192.168.201.136@tcp35 [11773.952021] LNet: Removed LNI 192.168.201.136@tcp36 [11773.953607] LNet: Removed LNI 192.168.201.136@tcp37 [11773.955561] LNet: Removed LNI 192.168.201.136@tcp38 [11773.958152] LNet: Removed LNI 192.168.201.136@tcp39 [11773.960637] LNet: Removed LNI 192.168.201.136@tcp40 [11773.962218] LNet: Removed LNI 192.168.201.136@tcp41 [11773.963831] LNet: Removed LNI 192.168.201.136@tcp42 [11773.965517] LNet: Removed LNI 192.168.201.136@tcp43 [11773.967641] LNet: Removed LNI 192.168.201.136@tcp44 [11773.969402] LNet: Removed LNI 192.168.201.136@tcp45 [11773.971806] LNet: Removed LNI 192.168.201.136@tcp46 [11773.974369] LNet: Removed LNI 192.168.201.136@tcp47 [11773.976269] LNet: Removed LNI 192.168.201.136@tcp48 [11773.978234] LNet: Removed LNI 192.168.201.136@tcp49 [11773.987494] LNet: Removed LNI 192.168.201.136@tcp50 [11774.311466] Key type .llcrypt unregistered [11774.312575] Key type ._llcrypt unregistered [11781.146695] Key type ._llcrypt registered [11781.147679] Key type .llcrypt registered [11781.188078] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [11787.899338] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11788.218184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11788.258391] alg: No test for adler32 (adler32-zlib) [11789.114820] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [11789.209777] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11790.791195] Key type lgssc registered [11791.142852] Lustre: Echo OBD driver; http://www.lustre.org/ [11802.442724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11802.448518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11803.530385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11803.540684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11803.580309] Lustre: lustre-MDT0000: new disk, initializing [11803.602354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11803.608540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11804.982470] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11809.023970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11809.068812] Lustre: 505173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11809.094629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11809.096722] Lustre: Skipped 1 previous similar message [11809.138860] Lustre: lustre-MDT0001: new disk, initializing [11809.175471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11809.187904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11809.192266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11810.504070] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11814.200134] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11816.134517] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11818.694394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11818.775623] Lustre: *** cfs_fail_loc=90f, val=0*** [11818.802923] Lustre: lustre-OST0000: new disk, initializing [11818.804516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11818.806889] Lustre: 507188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11818.835092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11820.754432] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11824.386242] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11826.169100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11826.171906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11826.209896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11831.269642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11831.270250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11831.273027] LustreError: Skipped 1 previous similar message [11831.274226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11831.281526] Lustre: Skipped 1 previous similar message [11832.881526] Lustre: server umount lustre-OST0000 complete [11834.847899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11834.850962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11834.855672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11834.857627] Lustre: Skipped 1 previous similar message [11836.384903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11836.385697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11840.345912] Lustre: server umount lustre-MDT0000 complete [11841.620311] LustreError: 505970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947038 with bad export cookie 2717144341385274027 [11841.623519] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11841.625532] LustreError: 505970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11841.744993] Lustre: server umount lustre-MDT0001 complete [11845.590051] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11846.612701] Key type lgssc unregistered [11846.736379] LNet: 509090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11846.738244] LNetError: 509090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11846.749303] LNet: Removed LNI 192.168.201.136@tcp [11847.045238] Key type .llcrypt unregistered [11847.046186] Key type ._llcrypt unregistered [11854.577547] Key type ._llcrypt registered [11854.578457] Key type .llcrypt registered [11854.611535] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11854.970719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11855.014565] alg: No test for adler32 (adler32-zlib) [11855.873929] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [11855.960703] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11857.535127] Key type lgssc registered [11857.913973] Lustre: Echo OBD driver; http://www.lustre.org/ [11860.594368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11860.605278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11861.771834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11863.172214] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11866.052544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.175853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11867.558057] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11869.866444] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11871.638290] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11873.842706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11873.947655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11875.827488] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11878.175570] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11892.515112] loop0: detected capacity change from 0 to 204800000 [11892.580356] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11892.584762] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11892.590053] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11894.892976] loop0: detected capacity change from 0 to 204800000 [11894.949501] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11894.952516] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11894.956168] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11896.942550] loop0: detected capacity change from 0 to 204800000 [11899.167430] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11899.220737] Lustre: fs15246-OST0000: new disk, initializing [11899.222503] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11899.225305] Lustre: 513948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11899.243319] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11901.396999] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11905.692987] loop1: detected capacity change from 0 to 204800000 [11907.783899] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11907.849930] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11907.886960] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11907.938304] Lustre: fs15246-MDT0000: new disk, initializing [11908.004678] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11908.021641] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11908.088243] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11909.153041] Lustre: 515069:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000262e5ad0 x1871652640030208/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784947122 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11909.449858] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11918.239391] LustreError: 515069:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [11918.244061] LustreError: 515069:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [11918.248314] LustreError: 515069:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [11919.844310] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11919.848659] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11923.349585] Lustre: server umount fs15246-MDT0000 complete [11924.963141] LustreError: 510332:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11924.968918] LustreError: 510332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11926.594483] Lustre: server umount fs15246-OST0000 complete [11933.664322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11933.664413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.667197] LustreError: Skipped 1 previous similar message [11933.667754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11933.672197] Lustre: Skipped 1 previous similar message [11933.677615] Lustre: Skipped 1 previous similar message [11935.203382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11935.207204] Lustre: Skipped 1 previous similar message [11936.426184] Lustre: server umount lustre-OST0000 complete [11940.322205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11940.330633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11940.334375] Lustre: Skipped 1 previous similar message [11944.215873] Lustre: server umount lustre-MDT0000 complete [11945.441399] LustreError: 510336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11945.447407] LustreError: 510336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11945.577544] LustreError: 512177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947142 with bad export cookie 8202017452529111057 [11945.580957] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11945.581662] LustreError: 512177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11945.704456] Lustre: server umount lustre-MDT0001 complete [11949.399463] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11950.522017] Key type lgssc unregistered [11950.658428] LNet: 518160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11950.660613] LNetError: 518160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11950.671387] LNet: Removed LNI 192.168.201.136@tcp [11951.051133] Key type .llcrypt unregistered [11951.052565] Key type ._llcrypt unregistered [11959.348131] Key type ._llcrypt registered [11959.350173] Key type .llcrypt registered [11959.390782] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11959.719640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11959.765741] alg: No test for adler32 (adler32-zlib) [11960.614921] Lustre: Lustre: Build Version: 2.17.54_207_ge7645ef [11960.701610] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11962.279200] Key type lgssc registered [11962.641418] Lustre: Echo OBD driver; http://www.lustre.org/ [11978.703377] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [11982.350950] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11994.664462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11994.670074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11995.745630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11995.753810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11995.789053] Lustre: lustre-MDT0000: new disk, initializing [11995.807968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11995.812836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11997.055780] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12001.264095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12001.305136] Lustre: 522596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12001.325876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12001.327445] Lustre: Skipped 1 previous similar message [12001.362815] Lustre: lustre-MDT0001: new disk, initializing [12001.390602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12001.400696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12001.403190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12002.610223] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12006.001117] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12007.633646] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12009.833534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12009.934630] Lustre: lustre-OST0000: new disk, initializing [12009.936464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12009.940783] Lustre: 524566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12009.970771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12012.090869] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12013.627204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12013.632240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12013.665038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12015.907691] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12017.217521] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12017.295350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12018.609605] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12018.673486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12021.387616] Lustre: server umount lustre-OST0000 complete [12023.776732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12023.781184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12023.782634] Lustre: Skipped 1 previous similar message [12028.896190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12028.897790] Lustre: Skipped 1 previous similar message [12033.017645] Lustre: server umount lustre-MDT0000 complete [12034.256221] LustreError: 523391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947231 with bad export cookie 4776865994563928980 [12034.259415] LustreError: 523391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12034.259977] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12044.839620] Lustre: server umount lustre-MDT0001 complete [12053.278318] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [12056.884749] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12066.572684] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11816 sec ========= 22:41:02 (1784947262) [12067.089733] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:41:02 (1784947262) === [12068.103991] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:41:04 (1784947264) === [12081.206548] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12082.145868] Key type lgssc unregistered [12082.265386] LNet: 531202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12082.267665] LNetError: 531202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12082.277301] LNet: Removed LNI 192.168.201.136@tcp [12082.682896] Key type .llcrypt unregistered [12082.685244] Key type ._llcrypt unregistered