[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468061814 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003380] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005026] kvm-guest: setup PV IPIs [ 0.008994] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011020] pid_max: default: 32768 minimum: 301 [ 0.012172] LSM: Security Framework initializing [ 0.013084] Yama: becoming mindful. [ 0.014082] SELinux: Initializing. [ 0.015114] *** VALIDATE selinux *** [ 0.028108] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032315] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035060] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036145] *** VALIDATE tmpfs *** [ 0.038467] *** VALIDATE proc *** [ 0.039000] *** VALIDATE cgroup *** [ 0.039000] *** VALIDATE cgroup2 *** [ 0.039310] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041093] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043000] Spectre V2 : User space: Vulnerable [ 0.043019] Speculative Store Bypass: Vulnerable [ 0.046000] debug: unmapping init [mem 0xffffffffa5a59000-0xffffffffa5a60fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047908] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048034] ... version: 2 [ 0.049026] ... bit width: 48 [ 0.050027] ... generic registers: 4 [ 0.051017] ... value mask: 0000ffffffffffff [ 0.052027] ... max period: 00007fffffffffff [ 0.053026] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.056355] rcu: Hierarchical SRCU implementation. [ 0.061066] smp: Bringing up secondary CPUs ... [ 0.063177] x86: Booting SMP configuration: [ 0.064199] .... node #0, CPUs: #1 #2 #3 [ 0.090132] smp: Brought up 1 node, 4 CPUs [ 0.092022] smpboot: Max logical packages: 1 [ 0.093024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147180] node 0 deferred pages initialised in 48ms [ 0.156344] devtmpfs: initialized [ 0.158344] x86/mm: Memory block size: 128MB [ 0.163219] gcov: version magic: 0x41383552 [ 0.167619] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.170103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.173542] pinctrl core: initialized pinctrl subsystem [ 0.181408] [ 0.182012] ************************************************************* [ 0.190020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191015] ** ** [ 0.198020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.202017] ** ** [ 0.207018] ** This means that this kernel is built to expose internal ** [ 0.212024] ** IOMMU data structures, which may compromise security on ** [ 0.214000] ** your system. ** [ 0.216013] ** ** [ 0.218013] ** If you see this message and you are not debugging the ** [ 0.221016] ** kernel, report this immediately to your vendor! ** [ 0.223015] ** ** [ 0.227019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.232079] ************************************************************* [ 0.236036] NET: Registered protocol family 16 [ 0.240664] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.249918] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.255089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.266057] cpuidle: using governor menu [ 0.270100] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273903] PCI: Using configuration type 1 for base access [ 0.277145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.289439] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.290000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295035] cryptd: max_cpu_qlen set to 1000 [ 0.297298] ACPI: Added _OSI(Module Device) [ 0.300031] ACPI: Added _OSI(Processor Device) [ 0.309022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.310000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.317200] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.333000] ACPI: Interpreter enabled [ 0.333000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.335019] ACPI: Using IOAPIC for interrupt routing [ 0.336171] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.342594] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.358000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.363083] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.369124] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.378180] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.389535] acpiphp: Slot [2] registered [ 0.394226] acpiphp: Slot [5] registered [ 0.397554] acpiphp: Slot [6] registered [ 0.401249] acpiphp: Slot [7] registered [ 0.404290] acpiphp: Slot [8] registered [ 0.407249] acpiphp: Slot [9] registered [ 0.416183] acpiphp: Slot [10] registered [ 0.421668] acpiphp: Slot [3] registered [ 0.426371] acpiphp: Slot [4] registered [ 0.430164] acpiphp: Slot [11] registered [ 0.432139] acpiphp: Slot [12] registered [ 0.436227] acpiphp: Slot [13] registered [ 0.438148] acpiphp: Slot [14] registered [ 0.439192] acpiphp: Slot [15] registered [ 0.446178] acpiphp: Slot [16] registered [ 0.449195] acpiphp: Slot [17] registered [ 0.453151] acpiphp: Slot [18] registered [ 0.456238] acpiphp: Slot [19] registered [ 0.459447] acpiphp: Slot [20] registered [ 0.462143] acpiphp: Slot [21] registered [ 0.463153] acpiphp: Slot [22] registered [ 0.469293] acpiphp: Slot [23] registered [ 0.476049] acpiphp: Slot [24] registered [ 0.480175] acpiphp: Slot [25] registered [ 0.482177] acpiphp: Slot [26] registered [ 0.489363] acpiphp: Slot [27] registered [ 0.495171] acpiphp: Slot [28] registered [ 0.500186] acpiphp: Slot [29] registered [ 0.502137] acpiphp: Slot [30] registered [ 0.504303] acpiphp: Slot [31] registered [ 0.507935] PCI host bridge to bus 0000:00 [ 0.512034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.518038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.523037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.528030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.532033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.535036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.541219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.546559] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.550800] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.566019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.575070] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.578028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.580026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.586030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.591457] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.600723] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.609059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.617101] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 15625 usecs [ 0.623275] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.633023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.660024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.678091] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.691686] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.705023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.731042] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.758023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.780000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.807040] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.828024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.852024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.871000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.883022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.891022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.926000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.942100] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.951024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.959000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.983020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.997000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.006020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.016021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.035020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.044947] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.052023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.062021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.087020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.108375] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.118480] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.127451] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.128426] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.137519] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.153064] iommu: Default domain type: Passthrough [ 1.154000] SCSI subsystem initialized [ 1.160228] ACPI: bus type USB registered [ 1.165158] usbcore: registered new interface driver usbfs [ 1.173129] usbcore: registered new interface driver hub [ 1.178151] usbcore: registered new device driver usb [ 1.181471] pps_core: LinuxPPS API ver. 1 registered [ 1.185017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.192087] PTP clock support registered [ 1.198142] EDAC MC: Ver: 3.0.0 [ 1.202229] PCI: Using ACPI for IRQ routing [ 1.204984] NetLabel: Initializing [ 1.207017] NetLabel: domain hash size = 128 [ 1.208015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.211114] NetLabel: unlabeled traffic allowed by default [ 1.217070] vgaarb: loaded [ 1.222591] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.225016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.242632] clocksource: Switched to clocksource kvm-clock [ 1.716267] VFS: Disk quotas dquot_6.6.0 [ 1.717869] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.726759] *** VALIDATE ramfs *** [ 1.732566] *** VALIDATE hugetlbfs *** [ 1.738643] pnp: PnP ACPI init [ 1.751698] pnp: PnP ACPI: found 6 devices [ 1.811144] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.824940] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.834864] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.843557] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.849172] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.853485] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.864151] NET: Registered protocol family 2 [ 1.867096] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.913305] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.935399] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.956215] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.970692] TCP: Hash tables configured (established 65536 bind 65536) [ 1.987826] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 2.022190] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.030650] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.036645] NET: Registered protocol family 1 [ 2.044123] RPC: Registered named UNIX socket transport module. [ 2.058688] RPC: Registered udp transport module. [ 2.061443] RPC: Registered tcp transport module. [ 2.066725] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.080769] NET: Registered protocol family 44 [ 2.084806] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.094897] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.100283] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.105800] PCI: CLS 0 bytes, default 64 [ 2.113395] Unpacking initramfs... [ 2.488530] hrtimer: interrupt took 11652025 ns [ 6.746175] debug: unmapping init [mem 0xffff96aa7cc54000-0xffff96aa7ffbffff] [ 6.761958] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.765274] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 6.774693] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 9.205511] Initialise system trusted keyrings [ 9.208706] Key type blacklist registered [ 9.214345] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 9.254570] zbud: loaded [ 9.270861] *** VALIDATE nfs *** [ 9.275551] *** VALIDATE nfs4 *** [ 9.278681] pstore: using deflate compression [ 9.297381] Platform Keyring initialized [ 9.607809] NET: Registered protocol family 38 [ 9.612415] Key type asymmetric registered [ 9.614533] Asymmetric key parser 'x509' registered [ 9.619745] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 9.625594] io scheduler mq-deadline registered [ 9.628033] io scheduler kyber registered [ 9.630701] io scheduler bfq registered [ 9.638183] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 9.642339] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 9.646250] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 9.654358] ACPI: Power Button [PWRF] [ 9.661614] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 9.678969] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.702571] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 9.722284] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.759642] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.804779] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.848993] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.873586] Non-volatile memory driver v1.3 [ 9.879561] Linux agpgart interface v0.103 [ 10.036873] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 10.047184] vda: detected capacity change from 0 to 71311360 [ 10.082504] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 10.087795] vdb: detected capacity change from 0 to 1073741824 [ 10.129871] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 10.144552] vdc: detected capacity change from 0 to 2621440000 [ 10.180476] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 10.188842] vdd: detected capacity change from 0 to 2621440000 [ 10.227729] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.231790] vde: detected capacity change from 0 to 4294967296 [ 10.265975] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.272973] vdf: detected capacity change from 0 to 4294967296 [ 10.289718] libphy: Fixed MDIO Bus: probed [ 10.302511] usbcore: registered new interface driver usbserial_generic [ 10.306780] usbserial: USB Serial support registered for generic [ 10.310235] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 10.318876] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 10.321563] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 10.324758] mousedev: PS/2 mouse device common for all mice [ 10.334488] rtc_cmos 00:05: RTC can wake from S4 [ 10.338770] rtc_cmos 00:05: registered as rtc0 [ 10.341672] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 10.345226] intel_pstate: CPU model not supported [ 10.349584] hid: raw HID events driver (C) Jiri Kosina [ 10.352407] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 10.357981] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 10.362447] usbcore: registered new interface driver usbhid [ 10.362456] usbhid: USB HID core driver [ 10.362589] drop_monitor: Initializing network drop monitor service [ 10.362726] Initializing XFRM netlink socket [ 10.363410] NET: Registered protocol family 10 [ 10.364741] Segment Routing with IPv6 [ 10.364784] NET: Registered protocol family 17 [ 10.366169] mpls_gso: MPLS GSO support [ 10.386356] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 10.405344] RAS: Correctable Errors collector initialized. [ 10.408745] AVX version of gcm_enc/dec engaged. [ 10.411493] AES CTR mode by8 optimization enabled [ 10.565791] sched_clock: Marking stable (10565769146, 0)->(12569517177, -2003748031) [ 10.595355] registered taskstats version 1 [ 10.612780] Loading compiled-in X.509 certificates [ 10.615162] zswap: loaded using pool lzo/zbud [ 10.692914] Key type big_key registered [ 10.733839] Key type encrypted registered [ 10.735632] ima: No TPM chip found, activating TPM-bypass! [ 10.745545] ima: Allocated hash algorithm: sha1 [ 10.756180] ima: No architecture policies found [ 10.764438] evm: Initialising EVM extended attributes: [ 10.773444] evm: security.selinux [ 10.779654] evm: security.ima [ 10.784486] evm: security.capability [ 10.793040] evm: HMAC attrs: 0x1 [ 10.797946] rtc_cmos 00:05: setting system clock to 2026-06-16 04:14:20 UTC (1781583260) [ 10.810554] debug: unmapping init [mem 0xffffffffa6a03000-0xffffffffa6bfffff] [ 10.821657] debug: unmapping init [mem 0xffffffffa5782000-0xffffffffa5a58fff] [ 10.850554] Write protecting the kernel read-only data: 28672k [ 10.856523] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 10.861744] debug: unmapping init [mem 0xffffffffa4714000-0xffffffffa47fffff] [ 11.085836] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.122624] systemd[1]: Detected virtualization kvm. [ 11.130599] systemd[1]: Detected architecture x86-64. [ 11.138759] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 11.220613] systemd[1]: No hostname configured. [ 11.222779] systemd[1]: Set hostname to . [ 11.241794] random: systemd: uninitialized urandom read (16 bytes read) [ 11.256480] systemd[1]: Initializing machine ID from random generator. [ 11.472728] random: ln: uninitialized urandom read (6 bytes read) [ 11.754749] random: systemd: uninitialized urandom read (16 bytes read) [ 11.766590] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 11.773900] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 11.805611] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 14.043168] device-mapper: uevent: version 1.0.3 [ 14.050077] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 16.720313] virtio_net virtio0 ens2: renamed from eth0 [ 16.928429] random: fast init done [ 17.078492] scsi host0: ata_piix [ 17.125208] scsi host1: ata_piix [ 17.127051] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 17.144529] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 22.303190] random: crng init done [ 22.308505] random: 7 urandom warning(s) missed due to ratelimiting [ 24.728812] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 26.449415] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 29.765941] printk: systemd: 26 output lines suppressed due to ratelimiting [ 30.881142] SELinux: Disabled at runtime. [ 31.065484] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 31.088045] systemd[1]: Detected virtualization kvm. [ 31.095361] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 32.716713] systemd[1]: initrd-switch-root.service: Succeeded. [ 32.723159] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 32.741280] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 32.750398] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 32.757141] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 32.786905] systemd[1]: Starting Journal Service... Starting Journal Service... [ 32.815408] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 33.482638] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 34.393923] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.454291] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.636921] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 36.070281] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 36.146224] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 40.750113] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 41.542212] NFS: Registering the id_resolver key type [ 41.544719] Key type id_resolver registered [ 41.546513] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg611-server login: [ 120.073755] libcfs: loading out-of-tree module taints kernel. [ 120.240256] Key type ._llcrypt registered [ 120.242126] Key type .llcrypt registered [ 120.490989] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 147.824367] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 149.999667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 150.039449] alg: No test for adler32 (adler32-zlib) [ 151.861793] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 153.262610] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 155.161155] Key type lgssc registered [ 157.646889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 181.831859] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 235.653955] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 253.826212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 253.875438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 255.443047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 255.558194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 255.762841] Lustre: lustre-MDT0000: new disk, initializing [ 255.980994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.006311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 262.508502] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 280.206417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.427384] Lustre: 6520:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 280.492440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 280.499681] Lustre: Skipped 1 previous similar message [ 280.666628] Lustre: lustre-MDT0001: new disk, initializing [ 280.795086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 280.867380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 280.888573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 288.892574] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.509359] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 311.397347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 311.778917] Lustre: lustre-OST0000: new disk, initializing [ 311.790879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 311.804877] Lustre: 8459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 311.929084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 320.064962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 320.073855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 320.184542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 320.941278] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 340.311841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 340.475622] Lustre: lustre-OST0001: new disk, initializing [ 340.482079] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 340.493556] Lustre: 9536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 340.613240] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 346.777317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 346.787390] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 346.914035] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 349.039485] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 365.265828] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 373.253949] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 381.737988] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing check_logdir /tmp/testlogs/ [ 388.237455] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing yml_node [ 394.474769] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 398.098832] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 401.722383] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 403.624370] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 16 00:20:52 EDT 2026 [ 423.686836] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 425.780880] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 444.389852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.392879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 444.415746] Lustre: Skipped 1 previous similar message [ 444.428604] Lustre: Skipped 3 previous similar messages [ 449.519878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.529582] Lustre: Skipped 2 previous similar messages [ 454.628431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 454.641977] Lustre: Skipped 4 previous similar messages [ 457.184405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 457.476798] Lustre: server umount lustre-MDT0000 complete [ 462.730514] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781583712 with bad export cookie 8006877113904404158 [ 462.752475] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.759967] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 464.864894] LustreError: 8762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.866481] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.876816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 464.896443] LustreError: 8762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 464.942126] Lustre: Skipped 3 previous similar messages [ 467.938656] LustreError: 8762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.968105] LustreError: 8762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 473.061235] LustreError: 8762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.061489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 473.093720] Lustre: Skipped 4 previous similar messages [ 473.094868] LustreError: 8762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 477.152209] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 477.493721] Lustre: server umount lustre-MDT0001 complete [ 482.884606] Lustre: server umount lustre-OST0000 complete [ 487.863955] Lustre: server umount lustre-OST0001 complete [ 497.516845] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 509.663189] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 554.972746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.347249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 555.407955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 555.586148] Lustre: lustre-MDT0000: new disk, initializing [ 555.671412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.700695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 560.795578] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 574.483451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.727449] Lustre: 15097:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 574.803951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 574.809686] Lustre: Skipped 1 previous similar message [ 574.918406] Lustre: lustre-MDT0001: new disk, initializing [ 575.042011] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 575.107038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 575.122967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 580.808467] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 592.895972] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 599.098759] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 606.824209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 607.229971] Lustre: lustre-OST0000: new disk, initializing [ 607.238937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 607.254931] Lustre: 17067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 607.416938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.486975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 608.503230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 608.727896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 615.217144] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 629.102092] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 636.566202] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 637.148886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 642.864565] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 643.379499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 649.699753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 649.704633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.712771] Lustre: Skipped 1 previous similar message [ 649.717369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 649.724770] Lustre: Skipped 1 previous similar message [ 652.200283] Lustre: server umount lustre-OST0000 complete [ 659.950062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.972642] Lustre: Skipped 1 previous similar message [ 663.657679] Lustre: server umount lustre-MDT0000 complete [ 665.060770] LustreError: 15103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.091955] LustreError: 15103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 669.014717] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781583918 with bad export cookie 8006877113904405691 [ 669.016686] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.038835] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 669.693591] Lustre: server umount lustre-MDT0001 complete [ 674.959497] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:25:23 (1781583923) [ 678.054784] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 680.222681] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:25:28 (1781583928) [ 689.031614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.596702] LustreError: 19479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.720442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 695.715783] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 699.886976] LustreError: 19480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.904948] LustreError: 19480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 707.545211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.419085] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 723.878445] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 731.422513] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 740.042664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 747.524835] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 756.362465] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 770.534638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 770.535515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.548559] LustreError: Skipped 1 previous similar message [ 770.578899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 770.581973] Lustre: Skipped 2 previous similar messages [ 776.468955] Lustre: server umount lustre-OST0000 complete [ 780.773454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.791773] Lustre: Skipped 2 previous similar messages [ 795.104201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 795.439759] Lustre: server umount lustre-MDT0000 complete [ 796.145260] LustreError: 19480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 796.164591] LustreError: 19480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 800.348677] LustreError: 19460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584050 with bad export cookie 8006877113904406573 [ 800.363187] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 800.364973] LustreError: 19460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 800.830706] Lustre: server umount lustre-MDT0001 complete [ 813.991731] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 818.115796] Key type lgssc unregistered [ 818.608563] LNet: 23041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.620181] LNetError: 23041:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.650880] LNet: Removed LNI 192.168.206.111@tcp [ 820.273249] Key type .llcrypt unregistered [ 820.278884] Key type ._llcrypt unregistered [ 850.157382] Key type ._llcrypt registered [ 850.163248] Key type .llcrypt registered [ 850.325119] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 851.861593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 851.892097] alg: No test for adler32 (adler32-zlib) [ 853.121330] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 853.492782] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 855.320195] Key type lgssc registered [ 857.045252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.882435] loop: module loaded [ 867.984996] loop0: detected capacity change from 0 to 204800000 [ 868.243939] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 868.259672] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 868.977672] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 876.839547] loop0: detected capacity change from 0 to 204800000 [ 877.097335] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 877.121650] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 877.163909] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 884.240575] loop0: detected capacity change from 0 to 204800000 [ 890.850844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 890.874303] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.228770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 892.330438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 892.432649] Lustre: lustre-MDT0000: new disk, initializing [ 892.647431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.705192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 898.673485] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 909.319331] loop1: detected capacity change from 0 to 204800000 [ 915.330583] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 915.783641] Lustre: lustre-OST07c6: new disk, initializing [ 915.796760] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 915.813321] Lustre: Skipped 1 previous similar message [ 915.819529] Lustre: 26231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 915.960835] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 923.594992] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 926.363288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 926.389991] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 926.490113] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 930.409362] Lustre: 25216:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 930.428809] Lustre: 25216:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 930.440261] Lustre: 25216:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 930.453409] Lustre: 25216:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 930.463186] Lustre: 25216:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 930.469563] Lustre: 25216:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 946.600217] Lustre: 25216:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 946.610887] Lustre: 25216:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 946.624310] Lustre: 25216:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 946.638027] Lustre: 25216:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 946.648683] Lustre: 25216:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 946.660290] Lustre: 25216:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 951.555595] Lustre: 25216:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 951.572082] Lustre: 25216:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 951.594986] Lustre: 25216:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 951.604689] Lustre: 25216:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 951.613583] Lustre: 25216:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 951.624926] Lustre: 25216:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 951.636204] Lustre: 25216:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 951.650146] Lustre: 25216:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 951.664283] Lustre: 25216:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 951.674413] Lustre: 25216:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 951.684652] Lustre: 25216:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 951.696462] Lustre: 25216:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 954.954303] Lustre: 26914:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 954.963052] Lustre: 26914:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 954.974872] Lustre: 26914:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 954.984799] Lustre: 26914:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 954.992782] Lustre: 26914:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 954.998547] Lustre: 26914:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 955.006934] Lustre: 26914:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 955.016617] Lustre: 26914:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 955.024291] Lustre: 26914:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 955.031128] Lustre: 26914:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 955.037545] Lustre: 26914:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 955.047359] Lustre: 26914:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 959.858621] Lustre: 25216:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 959.877741] Lustre: 25216:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 959.885191] Lustre: 25216:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 959.891671] Lustre: 25216:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 959.900078] Lustre: 25216:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 959.913614] Lustre: 25216:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 959.924050] Lustre: 25216:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 959.934929] Lustre: 25216:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 959.943853] Lustre: 25216:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 959.952583] Lustre: 25216:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 959.960648] Lustre: 25216:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 959.966208] Lustre: 25216:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 969.575972] Lustre: 25217:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 969.584031] Lustre: 25217:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 969.589561] Lustre: 25217:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 969.595155] Lustre: 25217:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 969.599977] Lustre: 25217:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 969.604801] Lustre: 25217:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 969.610552] Lustre: 25217:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 969.614912] Lustre: 25217:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 969.623076] Lustre: 25217:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 969.627815] Lustre: 25217:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 969.633158] Lustre: 25217:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 969.637328] Lustre: 25217:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1023.461931] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.492528] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1028.587064] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1033.705945] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1035.232320] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1035.404378] Lustre: server umount lustre-OST07c6 complete [ 1046.200578] Lustre: server umount lustre-MDT0000 complete [ 1065.668287] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 1068.945368] Key type lgssc unregistered [ 1069.266980] LNet: 28789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.282316] LNetError: 28789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.305900] LNet: Removed LNI 192.168.206.111@tcp [ 1070.294896] Key type .llcrypt unregistered [ 1070.296742] Key type ._llcrypt unregistered [ 1094.134922] Key type ._llcrypt registered [ 1094.137987] Key type .llcrypt registered [ 1094.270473] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 1096.161439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1096.299289] alg: No test for adler32 (adler32-zlib) [ 1097.476836] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1097.704630] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 1099.408194] Key type lgssc registered [ 1101.033327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1110.580386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1110.632575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.309915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.413952] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1121.775568] LustreError: 30063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.801124] LustreError: 30063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1126.887341] LustreError: 30062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.075803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.637036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1133.782053] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1142.244663] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1148.552706] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1156.364284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.527851] LustreError: 31884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.727061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.243094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1164.255138] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1173.530475] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1191.469199] Lustre: server umount lustre-OST0000 complete [ 1199.076648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.087511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.091657] Lustre: Skipped 1 previous similar message [ 1204.198865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.210298] Lustre: Skipped 2 previous similar messages [ 1206.351806] Lustre: server umount lustre-MDT0000 complete [ 1209.327262] LustreError: 30790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.360722] LustreError: 30790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1210.336166] Lustre: 29373:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781584443/real 1781584443] req@0000000044abe404 x1868126588273408/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781584459 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1210.365367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.413938] LustreError: 30043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584460 with bad export cookie 4373370984144002012 [ 1210.420084] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.424246] LustreError: 30043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1210.723781] Lustre: server umount lustre-MDT0001 complete [ 1223.115356] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 1227.013427] Key type lgssc unregistered [ 1227.376936] LNet: 33799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.400519] LNetError: 33799:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.436793] LNet: Removed LNI 192.168.206.111@tcp [ 1228.886209] Key type .llcrypt unregistered [ 1228.888146] Key type ._llcrypt unregistered [ 1254.045432] Key type ._llcrypt registered [ 1254.051032] Key type .llcrypt registered [ 1254.282972] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 1255.584217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.685280] alg: No test for adler32 (adler32-zlib) [ 1256.778690] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1257.069686] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 1258.905698] Key type lgssc registered [ 1260.873844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1271.249790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1271.276358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.113467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.881264] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1282.536962] LustreError: 35069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.593958] LustreError: 35069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1287.659568] LustreError: 35070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.371452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.065716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1298.165112] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1309.533154] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1317.758874] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1326.251112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.987235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.332917] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1342.322505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1343.632595] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1356.773352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1356.785261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.805676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1357.808144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1357.822724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1357.857219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.083362] Lustre: server umount lustre-OST0000 complete [ 1368.049110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1368.049446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.068216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1368.082868] Lustre: Skipped 1 previous similar message [ 1368.096506] Lustre: Skipped 2 previous similar messages [ 1372.509827] Lustre: server umount lustre-MDT0000 complete [ 1373.174242] LustreError: 35064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1373.238880] LustreError: 35064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1378.273929] LustreError: 35799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.299826] LustreError: 35799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1379.132958] LustreError: 37566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584628 with bad export cookie 13708166163262910695 [ 1379.136302] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1379.140588] LustreError: 37566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1379.755121] Lustre: server umount lustre-MDT0001 complete [ 1395.403833] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 1399.548224] Key type lgssc unregistered [ 1400.005645] LNet: 38611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.015080] LNetError: 38611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.044639] LNet: Removed LNI 192.168.206.111@tcp [ 1401.145267] Key type .llcrypt unregistered [ 1401.150543] Key type ._llcrypt unregistered [ 1428.872780] Key type ._llcrypt registered [ 1428.876865] Key type .llcrypt registered [ 1429.019612] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 1430.146486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1430.180175] alg: No test for adler32 (adler32-zlib) [ 1431.390365] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1431.665359] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 1433.408873] Key type lgssc registered [ 1434.694150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1443.792441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1443.814258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.346464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1450.458702] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1455.587409] LustreError: 40030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.605795] LustreError: 40030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1460.440262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.706513] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1460.920321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1466.315698] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1473.699976] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1482.358380] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1491.197698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.668274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.862671] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1508.653200] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1516.299803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1525.002266] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.11@tcp (stopping) [ 1526.765807] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.774301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.792469] Lustre: Skipped 2 previous similar messages [ 1526.808580] Lustre: Skipped 2 previous similar messages [ 1527.154699] Lustre: server umount lustre-MDT0000 complete [ 1530.076332] LustreError: 40027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.567377] LustreError: 40009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584781 with bad export cookie 17079295132657185996 [ 1531.578987] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1531.582539] LustreError: 40009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1531.880896] LustreError: 40025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.887719] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.927150] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1535.214421] LustreError: 40026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.994277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1537.001883] Lustre: Skipped 1 previous similar message [ 1537.924242] Lustre: server umount lustre-MDT0001 complete [ 1568.224199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1568.419403] Lustre: server umount lustre-OST0000 complete [ 1588.998781] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 1592.083207] Key type lgssc unregistered [ 1592.374946] LNet: 43917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1592.391851] LNetError: 43917:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.405742] LNet: Removed LNI 192.168.206.111@tcp [ 1593.348235] Key type .llcrypt unregistered [ 1593.354327] Key type ._llcrypt unregistered [ 1620.832254] Key type ._llcrypt registered [ 1620.839419] Key type .llcrypt registered [ 1621.010604] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 1622.439507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1622.758862] alg: No test for adler32 (adler32-zlib) [ 1624.092794] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1624.430117] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 1626.241102] Key type lgssc registered [ 1627.461845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1636.005959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1636.034564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.752878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.759437] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1648.097842] LustreError: 45334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.123896] LustreError: 45334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1652.227273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.825452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1658.185142] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1666.429697] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1673.120708] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1682.110989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1682.407805] LustreError: 47156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.613301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.070709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1690.008497] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1698.851926] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1706.734980] Lustre: lustre-OST0000: Not available for connect from 192.168.206.11@tcp (stopping) [ 1708.523672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.529155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.536796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1708.543746] Lustre: Skipped 1 previous similar message [ 1708.571583] LustreError: Skipped 1 previous similar message [ 1711.843186] Lustre: lustre-OST0000: Not available for connect from 192.168.206.11@tcp (stopping) [ 1711.852052] Lustre: Skipped 1 previous similar message [ 1712.460570] Lustre: server umount lustre-OST0000 complete [ 1734.114672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.137127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.149759] Lustre: Skipped 1 previous similar message [ 1738.188422] Lustre: server umount lustre-MDT0000 complete [ 1739.237845] LustreError: 45333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1739.275511] LustreError: 45333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1743.484982] LustreError: 45313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584993 with bad export cookie 3911464706325147426 [ 1743.489408] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.494808] LustreError: 45313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1750.149969] Lustre: server umount lustre-MDT0001 complete [ 1763.507159] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 1766.927824] Key type lgssc unregistered [ 1767.337580] LNet: 49047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.352530] LNetError: 49047:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.371397] LNet: Removed LNI 192.168.206.111@tcp [ 1768.224663] Key type .llcrypt unregistered [ 1768.227660] Key type ._llcrypt unregistered [ 1796.169239] Key type ._llcrypt registered [ 1796.176362] Key type .llcrypt registered [ 1796.327393] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 1797.475192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.612397] alg: No test for adler32 (adler32-zlib) [ 1798.882516] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1799.124991] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 1800.860350] Key type lgssc registered [ 1802.311515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1811.215138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1811.234621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.755190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.981392] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1823.204742] LustreError: 50466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1823.231369] LustreError: 50466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1828.072524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.328489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1828.607023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.359806] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1845.051538] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1852.499641] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1860.408626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1860.870349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.750742] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1877.774814] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1882.496478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1890.096782] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1893.674765] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1906.094490] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-16 0h45m54s [ 1907.884945] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1910.515996] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.11@tcp (stopping) [ 1910.760413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1910.777504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.585273] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.11@tcp (stopping) [ 1911.591519] Lustre: Skipped 7 previous similar messages [ 1916.598408] Lustre: server umount lustre-MDT0000 complete [ 1916.637834] LustreError: 53177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.446250] LustreError: 53177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.462679] LustreError: 53177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1920.128169] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1922.782840] LustreError: 50461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.315022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.446517] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.719335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.781235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1927.741381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1930.769473] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 1934.538818] Lustre: DEBUG MARKER: Wait for df (36546) ... [ 1936.230580] Lustre: DEBUG MARKER: done [ 1941.989784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1941.999528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.009330] Lustre: Skipped 2 previous similar messages [ 1942.013995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1942.022864] Lustre: Skipped 3 previous similar messages [ 1943.015285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.685735] Lustre: server umount lustre-OST0000 complete [ 1953.254386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.267729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1953.280088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.302233] Lustre: Skipped 3 previous similar messages [ 1958.824697] Lustre: server umount lustre-MDT0000 complete [ 1960.423098] LustreError: 50466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.440237] LustreError: 50466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1963.123629] LustreError: 53116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585212 with bad export cookie 15890156836107496088 [ 1963.138404] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1963.598761] Lustre: server umount lustre-MDT0001 complete [ 1974.931266] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 1978.088684] Key type lgssc unregistered [ 1978.465051] LNet: 55412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.474923] LNetError: 55412:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.527550] LNet: Removed LNI 192.168.206.111@tcp [ 1979.486189] Key type .llcrypt unregistered [ 1979.491811] Key type ._llcrypt unregistered [ 2019.854689] Key type ._llcrypt registered [ 2019.863430] Key type .llcrypt registered [ 2019.985227] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 2021.338060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2021.368865] alg: No test for adler32 (adler32-zlib) [ 2022.501962] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2022.783127] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 2024.483508] Key type lgssc registered [ 2025.567503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2032.883727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2032.916151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.349303] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2034.402752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2034.706459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.438484] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2048.181870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.309402] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2048.337099] Lustre: 57621:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2048.659731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2053.755864] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2061.322947] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2067.553597] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2074.737602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2074.895765] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2075.095994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.405039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2082.181796] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2089.651709] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2096.913406] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-16 0h49m05s [ 2099.358525] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2107.301783] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2110.120928] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2110.163812] LustreError: 57628:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2130.741206] Lustre: lustre-MDT0000: Client e24981fe-705d-4b74-9df1-6bac216ddc65 (at 192.168.206.11@tcp) reconnecting [ 2150.193612] LustreError: 57628:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2153.169370] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2156.432280] Lustre: DEBUG MARKER: done [ 2167.276294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2167.293191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.318428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2169.529440] Lustre: server umount lustre-OST0000 complete [ 2176.488903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2176.511638] LustreError: Skipped 1 previous similar message [ 2176.521652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.535033] Lustre: Skipped 1 previous similar message [ 2176.543658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.556787] Lustre: Skipped 1 previous similar message [ 2177.653658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.664823] Lustre: Skipped 2 previous similar messages [ 2180.536496] Lustre: server umount lustre-MDT0000 complete [ 2184.246392] LustreError: 58364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585433 with bad export cookie 4284937544808336506 [ 2184.251045] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2184.257571] LustreError: 58364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2184.756772] Lustre: server umount lustre-MDT0001 complete [ 2195.780304] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 2198.677508] Key type lgssc unregistered [ 2199.021169] LNet: 61950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.035578] LNetError: 61950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.064593] LNet: Removed LNI 192.168.206.111@tcp [ 2199.927481] Key type .llcrypt unregistered [ 2199.932808] Key type ._llcrypt unregistered [ 2245.811938] Key type ._llcrypt registered [ 2245.813862] Key type .llcrypt registered [ 2246.022167] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 2247.377867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2247.632655] alg: No test for adler32 (adler32-zlib) [ 2248.830257] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2249.115282] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 2250.842237] Key type lgssc registered [ 2252.484403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2256.623681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2256.656047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.079983] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2258.122695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2258.473821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.609517] Lustre: Failing over lustre-MDT0000 [ 2259.931814] Lustre: server umount lustre-MDT0000 complete [ 2266.986433] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:51:55 (1781585515) [ 2274.651048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.222556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.718923] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2289.926955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.050924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2290.099738] Lustre: 64750:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2290.339199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2295.418980] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2303.501578] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2308.965726] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2316.021826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2316.277150] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2316.489995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.514474] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2324.108633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2330.313692] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2337.101707] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2344.930070] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.934407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2344.950704] Lustre: Skipped 2 previous similar messages [ 2344.955652] Lustre: Skipped 1 previous similar message [ 2350.054482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2350.065666] Lustre: Skipped 3 previous similar messages [ 2350.583880] Lustre: server umount lustre-MDT0000 complete [ 2354.452234] LustreError: 64745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585604 with bad export cookie 12856076315399873986 [ 2354.454903] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.464284] LustreError: 64745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2354.810449] Lustre: server umount lustre-MDT0001 complete [ 2358.554193] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2362.952310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2370.529108] Lustre: 63677:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781585604/real 1781585604] req@000000001ee0e09d x1868127795191936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781585620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2370.551391] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.881906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.304162] Lustre: 63678:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781585608/real 1781585608] req@000000002859c07a x1868127795192064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781585624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2375.648296] Lustre: 63675:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781585609/real 1781585609] req@00000000f4c27779 x1868127795192192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781585625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2379.818520] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb269f54b60b8eea5 [ 2379.835729] Lustre: MGC192.168.206.111@tcp: Connection restored to 0@lo (at 0@lo) [ 2380.362283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.532302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2381.344312] Lustre: 63676:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781585614/real 1781585614] req@00000000df84375c x1868127795192448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781585630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2385.422280] Lustre: 63677:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781585619/real 1781585619] req@0000000024e64403 x1868127795192576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781585635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2386.404753] LustreError: 68524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.452223] LustreError: 68524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2386.810931] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2391.523205] LustreError: 68525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.550717] LustreError: 68525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2396.653043] LustreError: 66602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.688305] LustreError: 66602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2396.716840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.014557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2400.118060] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2401.629756] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2409.259784] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2415.384647] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2425.841218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.846591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.847942] Lustre: Skipped 2 previous similar messages [ 2425.856991] Lustre: Skipped 2 previous similar messages [ 2429.361679] Lustre: server umount lustre-MDT0000 complete [ 2430.952076] LustreError: 68524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.976252] LustreError: 68524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2433.773993] LustreError: 67464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585683 with bad export cookie 12856076315399876261 [ 2433.782852] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2433.790056] LustreError: 67464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2434.170396] Lustre: server umount lustre-MDT0001 complete [ 2440.057621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2450.312877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.426040] Lustre: 63678:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781585685/real 1781585685] req@00000000b9e906d7 x1868127795220480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781585701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2451.465855] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.028898] LustreError: 66601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.186619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2459.260389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2464.271641] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2467.300172] LustreError: 71208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.332260] LustreError: 71208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2474.098828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.116830] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2480.130757] Lustre: Skipped 1 previous similar message [ 2480.333465] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2488.839808] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2494.853674] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2502.007166] Lustre: server umount lustre-MDT0000 complete [ 2505.698499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2505.722759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.741686] LustreError: 66601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.774269] LustreError: 66601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2506.336773] LustreError: 64744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585756 with bad export cookie 12856076315399877640 [ 2506.350752] LustreError: 64744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2506.354568] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2507.047847] Lustre: server umount lustre-MDT0001 complete [ 2513.655821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2517.142367] Lustre: DEBUG MARKER: files compared the same [ 2519.595465] Lustre: server umount lustre-OST0000 complete [ 2537.238154] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 2540.312138] Key type lgssc unregistered [ 2540.587754] LNet: 74481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2540.608440] LNetError: 74481:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.636234] LNet: Removed LNI 192.168.206.111@tcp [ 2541.680618] Key type .llcrypt unregistered [ 2541.687303] Key type ._llcrypt unregistered [ 2567.991462] Key type ._llcrypt registered [ 2567.999735] Key type .llcrypt registered [ 2568.139562] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 2569.163776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.283595] alg: No test for adler32 (adler32-zlib) [ 2570.429518] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2570.749513] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 2572.472293] Key type lgssc registered [ 2573.943908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2582.186532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2582.200838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.946310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.174930] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug +malloc all [ 2594.280185] LustreError: 75899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.310192] LustreError: 75899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2598.497353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.906887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2604.365158] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug +malloc all [ 2613.027767] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2619.338509] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2626.893921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2627.232825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.322631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2634.678482] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug +malloc all [ 2642.258275] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2654.191100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.191988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2654.209874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2656.482675] Lustre: server umount lustre-OST0000 complete [ 2664.421342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.430677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.449120] Lustre: Skipped 2 previous similar messages [ 2664.464508] Lustre: Skipped 2 previous similar messages [ 2667.022817] Lustre: server umount lustre-MDT0000 complete [ 2669.537401] LustreError: 75900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.558536] LustreError: 75900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2670.703958] LustreError: 75879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585920 with bad export cookie 9477849997477776924 [ 2670.714699] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2670.724390] LustreError: 75879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2671.124667] Lustre: server umount lustre-MDT0001 complete [ 2683.820884] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 2687.069842] Key type lgssc unregistered [ 2687.508668] LNet: 79432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.536188] LNetError: 79432:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2687.558657] LNet: Removed LNI 192.168.206.111@tcp [ 2688.455525] Key type .llcrypt unregistered [ 2688.463885] Key type ._llcrypt unregistered [ 2714.956320] Key type ._llcrypt registered [ 2714.958880] Key type .llcrypt registered [ 2715.120927] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 2716.320019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2716.396373] alg: No test for adler32 (adler32-zlib) [ 2717.494876] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2717.751614] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 2719.504602] Key type lgssc registered [ 2720.964266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2728.829824] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2728.864216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.481353] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2749.600299] LustreError: 80825:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.111@tcp: failed processing log, type 4: rc = -110 [ 2776.417735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2783.098987] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2791.025886] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2799.229054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.414497] Lustre: *** cfs_fail_loc=706, val=0*** [ 2799.860771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.962133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2805.114630] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2805.226057] LustreError: 80850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.278506] LustreError: 80850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2810.352325] LustreError: 81910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.384036] LustreError: 81910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2815.464507] LustreError: 81911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.487094] LustreError: 81911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2816.088132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.708523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2822.620566] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2831.609561] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2838.258862] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2841.058711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2841.071878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.093802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.086681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2842.102861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.120604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.100691] Lustre: server umount lustre-OST0000 complete [ 2852.015378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.062869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.083324] Lustre: Skipped 2 previous similar messages [ 2857.447168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.464940] Lustre: Skipped 1 previous similar message [ 2858.746423] Lustre: server umount lustre-MDT0000 complete [ 2862.566804] LustreError: 81915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.645401] LustreError: 81915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2863.849825] LustreError: 80832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586113 with bad export cookie 3417041240501923528 [ 2863.890092] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2864.450454] Lustre: server umount lustre-MDT0001 complete [ 2879.501868] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 2882.906358] Key type lgssc unregistered [ 2883.258008] LNet: 84455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2883.271320] LNetError: 84455:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2883.298251] LNet: Removed LNI 192.168.206.111@tcp [ 2884.243880] Key type .llcrypt unregistered [ 2884.250546] Key type ._llcrypt unregistered [ 2912.605525] Key type ._llcrypt registered [ 2912.609637] Key type .llcrypt registered [ 2912.793474] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 2913.871325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2914.187548] alg: No test for adler32 (adler32-zlib) [ 2915.252626] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2915.495535] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 2917.176413] Key type lgssc registered [ 2918.482801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2926.190836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2926.209234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.355384] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2940.406459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.968878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.849129] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2946.026295] LustreError: 86559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.054972] LustreError: 86559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2951.142782] LustreError: 86562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.168385] LustreError: 86562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2955.356526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2955.814892] LustreError: 86558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2955.832200] LustreError: 86558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2955.861414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2960.867520] LustreError: 86563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.899819] LustreError: 86563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2962.501185] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2965.990441] LustreError: 86562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.020492] LustreError: 86562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2972.327676] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.339592] Lustre: MGS: Client c91f4d2e-c3e6-4934-8f84-be3602de9760 (at 0@lo) reconnecting [ 2972.386058] LustreError: 86573:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2972.689594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2972.821134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2978.523495] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 2998.242604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.259739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2998.274846] Lustre: Skipped 1 previous similar message [ 2998.322252] Lustre: Skipped 1 previous similar message [ 3001.101225] Lustre: server umount lustre-OST0000 complete [ 3008.485516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3008.486612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.497254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3008.504320] Lustre: Skipped 1 previous similar message [ 3011.687649] Lustre: server umount lustre-MDT0000 complete [ 3013.602897] LustreError: 86563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.616229] LustreError: 86563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3016.191844] Lustre: server umount lustre-MDT0001 complete [ 3020.372811] Lustre: server umount MGS complete [ 3033.571582] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:04:42 (1781586282) [ 3065.746579] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 3076.197666] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 3118.789095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.333335] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3134.885454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.355271] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3135.401295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3135.498841] Lustre: lustre-MDT0001: new disk, initializing [ 3135.613909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3135.713377] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3135.730407] Lustre: 93751:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3135.745306] LustreError: 93751:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3141.087171] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3152.067698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.332847] Lustre: lustre-OST0000: new disk, initializing [ 3152.336374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3152.342592] Lustre: 94517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3152.406902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.153615] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3160.161972] Lustre: 94867:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000037c20cc4 x1868128494594048/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781586425 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3169.918949] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3177.691573] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.701689] Lustre: MGS: Client b26e5ce3-1a61-444e-99b3-510286ed5dd0 (at 0@lo) reconnecting [ 3177.761975] Lustre: 95563:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3177.782616] LustreError: 93737:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3177.793495] LustreError: 93737:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3177.809531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3177.902760] Lustre: lustre-MDT0000: new disk, initializing [ 3178.047100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.075436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3178.098732] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3179.175666] Lustre: 95588:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000068688db7 x1868128494601472/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781586444 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3182.696945] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3184.290385] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3187.691869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3187.704375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3188.265495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3188.345418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3213.282499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3213.293831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.308564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3213.321662] Lustre: Skipped 1 previous similar message [ 3214.311527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.854797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3215.859437] Lustre: Skipped 2 previous similar messages [ 3218.796546] Lustre: server umount lustre-OST0000 complete [ 3226.095564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3226.109924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.119109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3226.132416] Lustre: Skipped 2 previous similar messages [ 3226.148246] Lustre: Skipped 1 previous similar message [ 3228.876459] Lustre: server umount lustre-MDT0000 complete [ 3231.202607] LustreError: 96016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.215628] LustreError: 96016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3247.072379] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3247.449101] Lustre: server umount lustre-MDT0001 complete [ 3252.547746] Lustre: server umount MGS complete [ 3265.770679] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:08:34 (1781586514) [ 3287.302100] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 3291.125895] Key type lgssc unregistered [ 3291.514870] LNet: 98090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3291.524495] LNetError: 98090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3291.550395] LNet: Removed LNI 192.168.206.111@tcp [ 3292.418776] Key type .llcrypt unregistered [ 3292.421358] Key type ._llcrypt unregistered [ 3311.888803] Key type ._llcrypt registered [ 3311.892754] Key type .llcrypt registered [ 3312.021162] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 3313.241792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3313.457696] alg: No test for adler32 (adler32-zlib) [ 3314.529691] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 3314.868034] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 3316.600165] Key type lgssc registered [ 3317.872299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3326.867951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3326.901586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.454519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.008376] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3338.727273] LustreError: 99312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.761349] LustreError: 99312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3343.846509] LustreError: 99313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3344.515269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.341621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.575319] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3361.416255] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3369.175769] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3377.551693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.989715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.903843] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3395.448593] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3398.525172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3410.914460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3410.926041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.947715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3413.986389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.989858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.517051] Lustre: server umount lustre-OST0000 complete [ 3422.177483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3422.188252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.208880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.211427] Lustre: Skipped 1 previous similar message [ 3424.226293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.226681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.293726] Lustre: server umount lustre-MDT0000 complete [ 3429.345728] LustreError: 99312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.367309] LustreError: 99312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3430.802919] LustreError: 99294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586680 with bad export cookie 9203248721606514368 [ 3430.813719] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.819765] LustreError: 99294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3431.277684] Lustre: server umount lustre-MDT0001 complete [ 3446.681690] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 3450.839293] Key type lgssc unregistered [ 3451.284878] LNet: 102872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3451.295781] LNetError: 102872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3451.326728] LNet: Removed LNI 192.168.206.111@tcp [ 3452.936444] Key type .llcrypt unregistered [ 3452.939779] Key type ._llcrypt unregistered [ 3475.816463] Key type ._llcrypt registered [ 3475.818755] Key type .llcrypt registered [ 3475.929608] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 3477.096040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3477.320127] alg: No test for adler32 (adler32-zlib) [ 3478.665584] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 3479.211643] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 3481.140580] Key type lgssc registered [ 3483.070248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3492.596403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3492.608524] LustreError: 103969:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3492.648599] LustreError: 103976:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3492.651869] LustreError: 103969:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4969 [ 3492.660056] LustreError: 103969:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3492.664428] LustreError: 103969:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3492.677183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.492280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.988868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.371816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3508.893423] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3515.031342] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing lsmod [ 3518.427558] LustreError: 105178:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3518.460585] LustreError: 105182:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3518.466624] LustreError: 105178:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 3518.471554] LustreError: 105178:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3518.482312] LustreError: 105178:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3518.494893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3519.071855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.146881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3525.090318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3525.102298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.133890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.629924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.634396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.650758] Lustre: Skipped 1 previous similar message [ 3526.682710] Lustre: Skipped 2 previous similar messages [ 3528.746185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3528.765762] Lustre: Skipped 2 previous similar messages [ 3533.796011] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.809222] Lustre: Skipped 2 previous similar messages [ 3537.376276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3537.682171] Lustre: server umount lustre-MDT0000 complete [ 3538.919437] LustreError: 105218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.962124] LustreError: 105218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3543.180494] LustreError: 103993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586792 with bad export cookie 6460442325343124091 [ 3543.189117] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3543.197443] LustreError: 103993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3544.037078] LustreError: 104009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.047851] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.068778] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3545.185651] LustreError: 104445:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000093fded16 x1868129084924544/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3545.407770] Lustre: server umount lustre-MDT0001 complete [ 3550.161657] Lustre: server umount lustre-OST0000 complete [ 3561.740865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.618815] LustreError: 106248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3562.834336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3568.101799] LustreError: 106249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.356867] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3573.222526] LustreError: 106248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.449092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.197440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3590.822429] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3601.563071] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3610.186220] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3620.143233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.693027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.450528] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3634.142545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3640.873574] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3648.482123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3648.493989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3648.513163] LustreError: 108548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3648.539173] LustreError: 108548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3648.613472] Lustre: server umount lustre-OST0000 complete [ 3654.632185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.635799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3654.657193] Lustre: Skipped 1 previous similar message [ 3654.672234] Lustre: Skipped 1 previous similar message [ 3664.143807] Lustre: server umount lustre-MDT0000 complete [ 3664.869881] LustreError: 107149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3664.895231] LustreError: 107149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3668.591347] LustreError: 106228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586918 with bad export cookie 6460442325343125036 [ 3668.623825] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3669.125623] Lustre: server umount lustre-MDT0001 complete [ 3682.184493] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 3685.289986] Key type lgssc unregistered [ 3685.663482] LNet: 109786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3685.682681] LNetError: 109786:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3685.718087] LNet: Removed LNI 192.168.206.111@tcp [ 3687.131715] Key type .llcrypt unregistered [ 3687.133927] Key type ._llcrypt unregistered [ 3717.277513] Key type ._llcrypt registered [ 3717.283562] Key type .llcrypt registered [ 3717.473430] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 3718.862285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3718.888393] alg: No test for adler32 (adler32-zlib) [ 3720.054307] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 3720.387308] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 3722.096424] Key type lgssc registered [ 3723.296158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3732.008692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3732.022752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.634172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.861273] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3744.228389] LustreError: 111207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3744.246698] LustreError: 111207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3748.777973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.413102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3754.636299] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3763.647147] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3771.242115] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3780.591484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3781.268056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.324355] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3800.654113] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3803.781722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3813.245219] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3817.988693] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3824.609891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3824.619144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.623751] LustreError: Skipped 1 previous similar message [ 3824.665815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3824.683756] Lustre: Skipped 1 previous similar message [ 3829.731225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.749244] Lustre: Skipped 1 previous similar message [ 3834.852726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3834.863778] Lustre: Skipped 1 previous similar message [ 3837.411157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3837.732380] Lustre: server umount lustre-OST0000 complete [ 3845.093730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.097934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3845.109705] Lustre: Skipped 2 previous similar messages [ 3848.863677] Lustre: server umount lustre-MDT0000 complete [ 3850.210912] LustreError: 111207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3850.240754] LustreError: 111207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3853.759113] LustreError: 111186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587103 with bad export cookie 13179799546387095366 [ 3853.779442] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3853.793925] LustreError: 111186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3854.317829] Lustre: server umount lustre-MDT0001 complete [ 3869.846278] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 3873.787595] Key type lgssc unregistered [ 3874.249699] LNet: 114870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3874.270370] LNetError: 114870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3874.293767] LNet: Removed LNI 192.168.206.111@tcp [ 3875.420235] Key type .llcrypt unregistered [ 3875.423861] Key type ._llcrypt unregistered [ 3899.696471] Key type ._llcrypt registered [ 3899.702773] Key type .llcrypt registered [ 3899.988477] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 3902.139251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3902.216425] alg: No test for adler32 (adler32-zlib) [ 3903.350371] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 3903.741852] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 3905.520240] Key type lgssc registered [ 3907.602717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3918.675355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3918.699612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.424549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.133485] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3930.617571] LustreError: 116096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3930.662223] LustreError: 116096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3935.721476] LustreError: 116095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.440951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.138320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3947.830963] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3960.363852] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3967.915447] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3977.948286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3978.618967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3988.263866] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 3991.997050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3998.225613] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4010.465799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.482179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.497378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4012.514800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4012.537786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4012.540285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.388981] Lustre: server umount lustre-OST0000 complete [ 4022.246832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4022.253506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.292315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4022.314318] Lustre: Skipped 1 previous similar message [ 4026.689514] Lustre: server umount lustre-MDT0000 complete [ 4027.874872] LustreError: 116095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.919671] LustreError: 116095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4031.704107] LustreError: 116075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587281 with bad export cookie 12207405139661995124 [ 4031.707049] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4031.723499] LustreError: 116075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4032.532609] Lustre: server umount lustre-MDT0001 complete [ 4047.301548] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 4050.903323] Key type lgssc unregistered [ 4051.312199] LNet: 119632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4051.332799] LNetError: 119632:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4051.373668] LNet: Removed LNI 192.168.206.111@tcp [ 4052.755176] Key type .llcrypt unregistered [ 4052.759252] Key type ._llcrypt unregistered [ 4084.419675] Key type ._llcrypt registered [ 4084.422690] Key type .llcrypt registered [ 4084.579923] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 4086.110988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4086.432857] alg: No test for adler32 (adler32-zlib) [ 4087.737963] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 4088.148154] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 4089.888349] Key type lgssc registered [ 4091.535645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4102.268402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4102.295162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.092505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.556640] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4113.389379] LustreError: 121052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.404352] LustreError: 121052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4118.504984] LustreError: 121051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.667453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.202935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4126.783131] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4137.137930] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4144.897265] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4154.104550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4154.643324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4163.091282] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4173.488490] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4174.218824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 4181.585527] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4186.767176] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4186.772686] Lustre: Skipped 2 previous similar messages [ 4190.326802] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4190.369831] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4192.284653] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4192.291290] Lustre: Skipped 2 previous similar messages [ 4198.500967] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4198.505439] Lustre: Skipped 1 previous similar message [ 4205.415453] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.11@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4205.423871] Lustre: Skipped 2 previous similar messages [ 4206.426117] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.11@tcp 192.168.0.[2,10]@tcp [ 4219.751712] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.11/24@tcp in log lustre-MDT0000 [ 4219.770915] Lustre: Skipped 2 previous similar messages [ 4227.285498] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.11/24@tcp [ 4227.295299] Lustre: Skipped 1 previous similar message [ 4237.810657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4237.831481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.868741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4240.873575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4240.891652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.739479] Lustre: server umount lustre-OST0000 complete [ 4249.058119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4249.067360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4249.080222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4249.084348] Lustre: Skipped 1 previous similar message [ 4251.109535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.114152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4251.137268] Lustre: Skipped 1 previous similar message [ 4254.626862] Lustre: server umount lustre-MDT0000 complete [ 4256.231669] LustreError: 121051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.269065] LustreError: 121051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4258.893302] LustreError: 121031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587508 with bad export cookie 5406538434168222386 [ 4258.902333] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4258.909144] LustreError: 121031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4259.351861] Lustre: server umount lustre-MDT0001 complete [ 4274.776415] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 4279.048454] Key type lgssc unregistered [ 4279.560367] LNet: 125906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4279.580369] LNetError: 125906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4279.635496] LNet: Removed LNI 192.168.206.111@tcp [ 4280.999802] Key type .llcrypt unregistered [ 4281.005459] Key type ._llcrypt unregistered [ 4318.045591] Key type ._llcrypt registered [ 4318.048017] Key type .llcrypt registered [ 4318.187322] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 4319.219741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4319.256042] alg: No test for adler32 (adler32-zlib) [ 4320.599349] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 4320.976750] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 4322.704804] Key type lgssc registered [ 4324.094603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4333.459845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4333.488688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4334.962506] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4334.999931] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.11/24@tcp [ 4335.036788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.892434] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4345.315827] LustreError: 127470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4345.347337] LustreError: 127470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4350.444378] LustreError: 127471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.937803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.458444] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4351.496633] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.11/24@tcp [ 4351.537472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4357.024290] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4366.148427] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4371.987422] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4380.073972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4380.483773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4381.580075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 4387.763793] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4396.591672] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4412.387991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4412.389692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.394873] LustreError: Skipped 1 previous similar message [ 4412.400515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4412.419928] Lustre: Skipped 1 previous similar message [ 4412.444330] Lustre: Skipped 1 previous similar message [ 4417.510316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4417.522762] Lustre: Skipped 1 previous similar message [ 4422.112297] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4422.339355] Lustre: server umount lustre-OST0000 complete [ 4427.503979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.522660] Lustre: Skipped 1 previous similar message [ 4427.525758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.868747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.892633] Lustre: Skipped 2 previous similar messages [ 4435.962734] Lustre: server umount lustre-MDT0000 complete [ 4437.999119] LustreError: 127471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.037638] LustreError: 127471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4441.299316] LustreError: 130103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587691 with bad export cookie 8076294617190661851 [ 4441.307184] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4441.309441] LustreError: 130103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4441.692250] Lustre: server umount lustre-MDT0001 complete [ 4455.923400] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 4459.583357] Key type lgssc unregistered [ 4460.059082] LNet: 131060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4460.080318] LNetError: 131060:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4460.118084] LNet: Removed LNI 192.168.206.111@tcp [ 4461.422772] Key type .llcrypt unregistered [ 4461.444944] Key type ._llcrypt unregistered [ 4491.013458] Key type ._llcrypt registered [ 4491.015843] Key type .llcrypt registered [ 4491.167658] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 4508.969820] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 4510.583926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4510.638947] alg: No test for adler32 (adler32-zlib) [ 4511.895229] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 4512.190941] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 4513.928442] Key type lgssc registered [ 4515.481725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4557.695401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4557.728681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.132525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4559.163238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4559.291146] Lustre: lustre-MDT0000: new disk, initializing [ 4559.488149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4559.514479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4565.451473] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4578.868595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.094532] Lustre: 134463:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4579.211958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4579.216143] Lustre: Skipped 1 previous similar message [ 4579.318469] Lustre: lustre-MDT0001: new disk, initializing [ 4579.469908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4579.535745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4579.552701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4585.685774] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4596.995444] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4603.687676] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4612.338158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4612.821857] Lustre: lustre-OST0000: new disk, initializing [ 4612.834691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4612.848783] Lustre: 136432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4612.966075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4619.427774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4619.449917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4619.686353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4621.177737] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4633.116049] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4639.294723] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4639.585438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4645.957371] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4646.313972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4650.474834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4650.489686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4650.496772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4650.502974] Lustre: Skipped 1 previous similar message [ 4650.534666] LustreError: Skipped 1 previous similar message [ 4655.544338] Lustre: server umount lustre-OST0000 complete [ 4660.527620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4660.550769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4660.561743] Lustre: Skipped 2 previous similar messages [ 4665.830615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4665.839530] Lustre: Skipped 1 previous similar message [ 4667.145334] Lustre: server umount lustre-MDT0000 complete [ 4672.056341] LustreError: 134456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587921 with bad export cookie 15351424394824785603 [ 4672.070651] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.079067] LustreError: 134456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4672.562206] Lustre: server umount lustre-MDT0001 complete [ 4683.627634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4684.420172] LustreError: 138651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.443104] LustreError: 138651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4684.530501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.898841] LustreError: 138652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4690.730881] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4695.016537] LustreError: 138651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.144050] LustreError: 138652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4701.919033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.657907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4708.812833] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4718.692535] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4726.624744] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4736.485533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4736.992385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4745.926326] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4755.328089] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4760.974822] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4761.237698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4766.409439] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4766.610907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4777.257500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4777.537091] Lustre: lustre-OST0001: new disk, initializing [ 4777.541573] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4777.547654] Lustre: 142026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4777.715519] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4785.085743] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 4785.328395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4785.340557] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4785.553901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4794.812890] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4795.121860] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4801.563050] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4801.920000] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4810.211240] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94c585e33800.ost_server_uuid 50 [ 4812.517525] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff94c585e33800.ost_server_uuid in IDLE state after 0 sec [ 4826.597712] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4826.607306] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4826.611104] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4826.616760] Lustre: Skipped 2 previous similar messages [ 4826.636449] Lustre: Skipped 1 previous similar message [ 4830.702495] Lustre: server umount lustre-OST0001 complete [ 4831.722802] LustreError: 143330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4831.737992] LustreError: 143330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4836.330663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4836.338615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.339543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.350077] LustreError: Skipped 2 previous similar messages [ 4836.368127] Lustre: Skipped 1 previous similar message [ 4841.718357] Lustre: server umount lustre-OST0000 complete [ 4847.075285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.077600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4847.100911] Lustre: Skipped 2 previous similar messages [ 4847.116021] Lustre: Skipped 1 previous similar message [ 4852.748514] Lustre: server umount lustre-MDT0000 complete [ 4856.854147] LustreError: 138633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588106 with bad export cookie 15351424394824786485 [ 4856.858271] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4856.860877] LustreError: 138633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4857.428863] Lustre: server umount lustre-MDT0001 complete [ 4879.915949] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 4883.533888] Key type lgssc unregistered [ 4883.858598] LNet: 145032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4883.869680] LNetError: 145032:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4884.902341] LNet: Removed LNI 192.168.206.111@tcp [ 4885.882091] Key type .llcrypt unregistered [ 4885.890199] Key type ._llcrypt unregistered [ 4929.459888] Key type ._llcrypt registered [ 4929.462683] Key type .llcrypt registered [ 4929.556318] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 4947.486537] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 4948.866121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4948.921926] alg: No test for adler32 (adler32-zlib) [ 4950.183944] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 4950.537412] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 4952.240282] Key type lgssc registered [ 4953.640264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4996.878489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4996.922738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.545726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4998.583768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4998.743419] Lustre: lustre-MDT0000: new disk, initializing [ 4998.925921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.956488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5005.848618] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5019.500459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.656216] Lustre: 149198:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5019.739796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5019.751621] Lustre: Skipped 1 previous similar message [ 5019.877850] Lustre: lustre-MDT0001: new disk, initializing [ 5019.987440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5020.059690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5020.079636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5024.727415] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5036.729699] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5044.341268] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5053.847067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5054.289617] Lustre: lustre-OST0000: new disk, initializing [ 5054.294500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5054.301967] Lustre: 151166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5054.411418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.762349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5060.783590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5060.969628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5062.404985] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5073.773910] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5078.547675] Lustre: Failing over lustre-OST0000 [ 5078.748624] Lustre: server umount lustre-OST0000 complete [ 5095.904125] Lustre: 147421:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781588329/real 1781588329] req@00000000aa13dcd2 x1868130628439424/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1781588345 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5095.941333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.953535] Lustre: 147423:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781588331/real 1781588331] req@000000008d7c51bf x1868130628440576/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781588347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5097.982118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5098.843870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5099.214624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5099.235236] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5103.075606] Lustre: 147422:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781588336/real 1781588336] req@00000000a5e730eb x1868130628441600/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781588352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5103.106443] Lustre: 147422:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5105.328137] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5106.609937] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5107.363259] Lustre: 147421:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781588341/real 1781588341] req@00000000a38da382 x1868130628442496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781588357 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5107.388344] Lustre: 147421:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5107.504995] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5107.505201] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 5110.889720] Lustre: Failing over lustre-MDT0000 [ 5111.185335] Lustre: server umount lustre-MDT0000 complete [ 5111.520178] Lustre: 147423:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781588345/real 1781588345] req@0000000006a242fc x1868130628443648/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781588361 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5111.546879] Lustre: 147423:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5112.294269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5112.305250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.887297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5117.657197] LustreError: 150484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.672132] LustreError: 150484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5120.994747] LustreError: 150328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.007790] LustreError: 150328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5122.790982] LustreError: 150484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5127.881511] LustreError: 152713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5127.908059] LustreError: 152713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5132.074932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.128830] Lustre: 147423:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781588365/real 1781588365] req@00000000f593822a x1868130628456320/t0(0) o400->MGC192.168.206.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781588381 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5132.178220] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5132.195348] LustreError: 150484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.235194] LustreError: 150484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5141.478119] LustreError: 150484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5141.481471] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd8307a9451984677 [ 5141.489659] LustreError: 150484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5141.513665] Lustre: MGC192.168.206.111@tcp: Connection restored to 0@lo (at 0@lo) [ 5141.523455] Lustre: Skipped 1 previous similar message [ 5141.812065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.881707] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5143.268755] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5146.892855] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5147.143358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5147.168956] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5147.219802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5157.358546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.365175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5157.374652] Lustre: Skipped 2 previous similar messages [ 5161.997410] Lustre: server umount lustre-OST0000 complete [ 5167.585447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5167.607486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5167.607563] Lustre: Skipped 1 previous similar message [ 5167.623067] Lustre: Skipped 2 previous similar messages [ 5172.645121] Lustre: server umount lustre-MDT0000 complete [ 5172.707165] LustreError: 149204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.741492] LustreError: 149204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5176.938372] LustreError: 149188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588426 with bad export cookie 15578085888517228151 [ 5176.944198] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5176.957394] LustreError: 149188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5177.309731] Lustre: server umount lustre-MDT0001 complete [ 5189.682621] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 5193.627826] Key type lgssc unregistered [ 5193.919283] LNet: 154835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5193.927719] LNetError: 154835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5193.946772] LNet: Removed LNI 192.168.206.111@tcp [ 5195.078926] Key type .llcrypt unregistered [ 5195.083975] Key type ._llcrypt unregistered [ 5240.260840] Key type ._llcrypt registered [ 5240.266289] Key type .llcrypt registered [ 5240.512505] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 5242.718839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5242.749868] alg: No test for adler32 (adler32-zlib) [ 5244.050724] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 5244.337506] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 5246.040209] Key type lgssc registered [ 5247.299637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5258.499967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5258.517668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5260.156093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5266.200268] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5269.477161] LustreError: 156359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5269.512171] LustreError: 156359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5274.594824] LustreError: 156358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5276.739292] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5279.717674] LustreError: 156359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.735504] LustreError: 156387:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5280.754772] LustreError: 156387:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5280.763499] LustreError: 156387:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5281.089377] Lustre: server umount lustre-MDT0000 complete [ 5298.675811] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 5311.624766] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 5357.445577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.934547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5358.029692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5358.154884] Lustre: lustre-MDT0000: new disk, initializing [ 5358.280726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5358.306971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5363.289925] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5374.937815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.066680] Lustre: 159693:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5375.136402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5375.143695] Lustre: Skipped 1 previous similar message [ 5375.232560] Lustre: lustre-MDT0001: new disk, initializing [ 5375.343713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5375.415861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5375.423615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5380.714455] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5392.269832] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5398.834846] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5407.352511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5407.718114] Lustre: lustre-OST0000: new disk, initializing [ 5407.722477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5407.728988] Lustre: 161664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5407.821752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.185168] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5417.128066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5417.143975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5417.313986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5426.092814] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5431.806128] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5434.005773] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:44:42 (1781588682) [ 5696.996628] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.002670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5697.032183] Lustre: Skipped 2 previous similar messages [ 5700.662142] Lustre: server umount lustre-MDT0000 complete [ 5702.113133] LustreError: 162843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.123206] LustreError: 162843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5704.873181] LustreError: 160488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588954 with bad export cookie 3869796977765933699 [ 5704.873362] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5704.896886] LustreError: 160488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5707.233110] LustreError: 159701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5707.244627] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.274182] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5707.279111] Lustre: Skipped 2 previous similar messages [ 5709.281846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5711.255611] Lustre: server umount lustre-MDT0001 complete [ 5721.930985] Lustre: server umount lustre-OST0000 complete [ 5732.662410] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 5744.051858] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 5783.682652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.997964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5784.045703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5784.131898] Lustre: lustre-MDT0000: new disk, initializing [ 5784.254025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.281502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5789.676674] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5801.053603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.209057] Lustre: 166336:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5801.330889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5801.341742] Lustre: Skipped 1 previous similar message [ 5801.457671] Lustre: lustre-MDT0001: new disk, initializing [ 5801.565081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5801.620695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5801.634947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5807.017398] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5818.846325] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5826.514195] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5835.594884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5835.927727] Lustre: lustre-OST0000: new disk, initializing [ 5835.933048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5835.947070] Lustre: 168306:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5836.060528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.656415] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5845.050373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5845.060363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5845.202956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5857.187976] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5863.062740] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5863.365526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5868.770686] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5869.110529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5876.200835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5876.205071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.207855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5876.221028] LustreError: Skipped 1 previous similar message [ 5876.252200] Lustre: Skipped 1 previous similar message [ 5877.623867] Lustre: server umount lustre-OST0000 complete [ 5883.361453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5883.377222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.413797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5883.424258] Lustre: Skipped 1 previous similar message [ 5889.191807] Lustre: server umount lustre-MDT0000 complete [ 5891.555858] LustreError: 166347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5891.602308] LustreError: 166347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5893.557292] LustreError: 166329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781589143 with bad export cookie 3869796977766092529 [ 5893.562706] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5893.572321] LustreError: 166329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5894.079212] Lustre: server umount lustre-MDT0001 complete [ 5904.183607] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:52:32 (1781589152) [ 5923.917130] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 5934.707899] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 5974.324381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.583722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5974.621986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5974.683518] Lustre: lustre-MDT0000: new disk, initializing [ 5974.761178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5974.780699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5979.528496] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 5990.471490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.595979] Lustre: 173064:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5990.962541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5996.251285] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6006.140801] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6011.375619] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6018.269022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6018.640362] Lustre: lustre-OST0000: new disk, initializing [ 6018.643386] Lustre: Skipped 1 previous similar message [ 6018.647950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6018.654628] Lustre: Skipped 2 previous similar messages [ 6018.659793] Lustre: 175043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6025.489591] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6026.826137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6026.841177] Lustre: Skipped 1 previous similar message [ 6026.851834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6026.958237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6034.731810] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6042.595565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.606836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6042.612903] Lustre: Skipped 2 previous similar messages [ 6042.616681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6042.663738] Lustre: Skipped 3 previous similar messages [ 6048.023714] Lustre: server umount lustre-OST0000 complete [ 6052.839706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6057.442364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6057.449024] LustreError: Skipped 1 previous similar message [ 6059.099488] Lustre: server umount lustre-MDT0000 complete [ 6063.075308] LustreError: 173078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.084342] LustreError: 173078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6063.450610] LustreError: 173057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781589313 with bad export cookie 3869796977766093411 [ 6063.453885] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6063.459665] LustreError: 173057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6063.797911] Lustre: server umount lustre-MDT0001 complete [ 6072.255147] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:55:21 (1781589321) [ 6088.989628] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 6099.456760] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 6138.317320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6138.721435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6138.750865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6138.811865] Lustre: lustre-MDT0000: new disk, initializing [ 6138.900791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.908163] Lustre: Skipped 2 previous similar messages [ 6138.924294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6144.482656] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6157.555794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6157.833411] Lustre: 179366:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6158.355949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6158.355949] Lustre: cli-cli-lustre-MDT0000-osp-MDT0001: Allocated meta-sequence [0x0000000200000401-0x0000000200000402]:0:mdt] [ 6164.369466] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6175.027625] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6181.033286] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6190.340358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6190.804651] Lustre: 181347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6197.871550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6198.069298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6198.240440] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6209.257445] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6218.730774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6218.732772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.745550] LustreError: Skipped 1 previous similar message [ 6218.746721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.746727] Lustre: Skipped 5 previous similar messages [ 6218.788683] Lustre: Skipped 2 previous similar messages [ 6223.390798] Lustre: server umount lustre-OST0000 complete [ 6242.272467] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6242.457629] Lustre: server umount lustre-MDT0000 complete [ 6244.331261] LustreError: 179372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.348953] LustreError: 179372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6246.755401] LustreError: 179359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781589496 with bad export cookie 3869796977766094496 [ 6246.776478] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6246.783949] LustreError: 179359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6247.120570] Lustre: server umount lustre-MDT0001 complete [ 6261.213930] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 6264.698212] Key type lgssc unregistered [ 6265.188052] LNet: 183204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6265.196913] LNetError: 183204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6266.222151] LNet: Removed LNI 192.168.206.111@tcp [ 6267.421294] Key type .llcrypt unregistered [ 6267.424124] Key type ._llcrypt unregistered [ 6297.092604] Key type ._llcrypt registered [ 6297.095157] Key type .llcrypt registered [ 6297.226473] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 6298.962586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6299.019284] alg: No test for adler32 (adler32-zlib) [ 6300.307886] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6300.632201] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 6302.424204] Key type lgssc registered [ 6304.108328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6313.027337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6313.059416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.568653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6319.238880] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6324.716494] LustreError: 184626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6324.737600] LustreError: 184626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6328.770422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.285130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6334.329590] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6341.446908] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6347.710302] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6355.783659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6356.128710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.256876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6363.443936] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6372.797233] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6394.353579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.355558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6394.378543] Lustre: Skipped 1 previous similar message [ 6394.388781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.401423] Lustre: Skipped 1 previous similar message [ 6398.544838] Lustre: server umount lustre-OST0000 complete [ 6404.584699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.594472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.618575] Lustre: Skipped 1 previous similar message [ 6404.643845] Lustre: Skipped 1 previous similar message [ 6409.697651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6409.712307] Lustre: Skipped 1 previous similar message [ 6410.173153] Lustre: server umount lustre-MDT0000 complete [ 6414.826187] LustreError: 184625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6414.852223] LustreError: 184625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6415.079184] LustreError: 184605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781589664 with bad export cookie 17228355424034868099 [ 6415.085557] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6415.088387] LustreError: 184605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6415.855389] Lustre: server umount lustre-MDT0001 complete [ 6430.615664] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 6433.804419] Key type lgssc unregistered [ 6434.214870] LNet: 188195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6434.223909] LNetError: 188195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6434.246240] LNet: Removed LNI 192.168.206.111@tcp [ 6435.313667] Key type .llcrypt unregistered [ 6435.316313] Key type ._llcrypt unregistered [ 6466.559311] Key type ._llcrypt registered [ 6466.562408] Key type .llcrypt registered [ 6466.656798] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 6467.692367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6467.770018] alg: No test for adler32 (adler32-zlib) [ 6468.933653] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6469.247674] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 6471.008362] Key type lgssc registered [ 6472.437848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6481.570435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6481.610893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.130327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6488.244291] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6492.643537] LustreError: 189612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6492.671403] LustreError: 189612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6497.766213] LustreError: 189611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.967633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.590036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6504.033828] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6512.176770] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6518.385841] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6525.999695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.344502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6527.361849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6532.840726] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6541.479822] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6553.058420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.082276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.100560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6554.378565] Lustre: lustre-OST0000: Not available for connect from 192.168.206.11@tcp (stopping) [ 6555.115465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6555.123235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.073319] Lustre: server umount lustre-OST0000 complete [ 6569.352685] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff94c58d20f000.ost_server_uuid 50 [ 6572.065471] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6589.937712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.983640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.987330] Lustre: Skipped 2 previous similar messages [ 6594.106295] Lustre: server umount lustre-MDT0000 complete [ 6595.053643] LustreError: 189612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6595.091982] LustreError: 189612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6598.534191] LustreError: 189593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781589848 with bad export cookie 17115473919941409958 [ 6598.541872] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6598.553136] LustreError: 189593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6599.141480] Lustre: server umount lustre-MDT0001 complete [ 6608.995544] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:04:17 (1781589857) [ 6616.890059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.366598] LustreError: 193528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.389727] LustreError: 193528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6617.452445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6622.690458] LustreError: 193529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.708876] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6632.931105] LustreError: 193528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6632.963741] LustreError: 193528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6634.130914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.576121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6639.981311] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6648.371576] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6654.863966] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6663.476042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6663.984544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6671.998828] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6680.578974] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6687.552112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6688.832503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6689.100990] Lustre: lustre-OST0001: new disk, initializing [ 6689.111847] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6689.119918] Lustre: 196372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6689.283655] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6696.069300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6696.083347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6696.176840] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6696.425336] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6707.434621] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6711.795958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6711.801939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6711.834282] Lustre: Skipped 2 previous similar messages [ 6716.791946] Lustre: server umount lustre-OST0000 complete [ 6716.899619] LustreError: 195352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6716.918745] LustreError: 195352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6723.703645] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6723.922290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6729.274906] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6729.590743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6747.622298] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6747.635981] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6747.640303] Lustre: Skipped 1 previous similar message [ 6747.643265] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6747.688724] Lustre: Skipped 3 previous similar messages [ 6750.225904] Lustre: server umount lustre-OST0001 complete [ 6757.348610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6757.353543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.374039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6760.748679] Lustre: server umount lustre-MDT0000 complete [ 6762.984162] LustreError: 193529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6763.003743] LustreError: 193529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 6764.903922] LustreError: 193509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781590014 with bad export cookie 17115473919941411022 [ 6764.919233] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6764.923908] LustreError: 193509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6765.302583] Lustre: server umount lustre-MDT0001 complete [ 6793.299676] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:07:21 (1781590041) [ 6802.290869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6802.708199] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6802.783157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6803.207559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6808.495680] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6818.531559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.680532] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6818.729928] Lustre: 199814:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6823.913932] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6832.869705] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6839.021917] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6846.547128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6846.851154] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6854.174635] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6856.402460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6863.517786] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6870.829588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6870.921640] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6871.192276] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6871.201849] Lustre: Skipped 2 previous similar messages [ 6878.108351] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 6886.823812] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6890.996871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6891.003305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6891.017080] Lustre: Skipped 2 previous similar messages [ 6891.021865] Lustre: Skipped 1 previous similar message [ 6896.474276] Lustre: server umount lustre-OST0000 complete [ 6897.886393] LustreError: 203459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6916.586672] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6918.941983] Lustre: server umount lustre-OST0001 complete [ 6926.819537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6926.821068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.822342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6926.822349] Lustre: Skipped 6 previous similar messages [ 6926.850310] Lustre: Skipped 3 previous similar messages [ 6929.241424] Lustre: server umount lustre-MDT0000 complete [ 6932.846837] LustreError: 202700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781590182 with bad export cookie 17115473919941412436 [ 6932.856637] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6932.863733] LustreError: 202700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6933.190577] Lustre: server umount lustre-MDT0001 complete [ 6960.763818] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:10:09 (1781590209) [ 6981.641433] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 6993.823163] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 7034.309120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.544945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7034.591911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7034.681479] Lustre: lustre-MDT0000: new disk, initializing [ 7034.788876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.834938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7040.333691] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7052.843862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.099303] Lustre: 207918:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7053.147467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7053.154995] Lustre: Skipped 1 previous similar message [ 7053.220466] Lustre: lustre-MDT0001: new disk, initializing [ 7053.371313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7053.387306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7058.677714] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7068.937148] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7075.577690] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7083.524974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7083.968446] Lustre: lustre-OST0000: new disk, initializing [ 7083.973217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7083.979555] Lustre: 209888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7092.821683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7092.835103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7093.009874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7093.627178] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7105.198136] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7110.291795] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7110.653551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7115.281244] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7115.705619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7118.818130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7118.821160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.823066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7118.823073] Lustre: Skipped 1 previous similar message [ 7118.856021] Lustre: Skipped 1 previous similar message [ 7123.867682] Lustre: server umount lustre-OST0000 complete [ 7135.009632] Lustre: server umount lustre-MDT0000 complete [ 7139.228770] LustreError: 208723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781590388 with bad export cookie 17115473919941413997 [ 7139.235652] LustreError: 208723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7139.236859] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7139.596736] Lustre: server umount lustre-MDT0001 complete [ 7149.495528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.168199] LustreError: 212106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7150.189034] LustreError: 212106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 7156.605354] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7167.043490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.681293] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7183.874860] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7191.195398] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7200.810206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7209.143305] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7218.016897] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7223.516807] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7223.719505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7229.247639] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7229.491979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7233.506663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7233.515897] LustreError: Skipped 1 previous similar message [ 7246.304203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7246.622755] Lustre: server umount lustre-OST0000 complete [ 7254.431804] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7260.565191] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7290.819770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7291.516108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.526425] Lustre: Skipped 5 previous similar messages [ 7298.804623] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7300.723544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7300.736196] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7300.750712] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7300.787797] Lustre: Skipped 1 previous similar message [ 7307.500171] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7309.415634] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7316.450867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7316.460230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7316.478600] Lustre: Skipped 4 previous similar messages [ 7316.485822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7316.500916] LustreError: Skipped 2 previous similar messages [ 7316.530907] Lustre: Skipped 13 previous similar messages [ 7320.331638] Lustre: server umount lustre-OST0000 complete [ 7331.163669] Lustre: server umount lustre-MDT0000 complete [ 7335.739583] LustreError: 216222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781590585 with bad export cookie 17115473919941414879 [ 7335.750133] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.753692] LustreError: 216222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7336.231577] Lustre: server umount lustre-MDT0001 complete [ 7347.059084] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:16:35 (1781590595) [ 7354.322266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7359.963366] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7370.187909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7376.435441] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7385.270160] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7392.399724] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7400.761491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7408.804928] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7417.833785] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7423.177732] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7425.694201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7431.310478] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7431.535354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7438.876463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7439.033910] Lustre: lustre-OST0001: new disk, initializing [ 7439.040288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7439.047297] Lustre: 221469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7445.638132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7445.669606] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7445.815515] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7446.216391] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7456.215364] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7461.424392] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7461.681777] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7466.754064] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7467.000837] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7471.586136] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7471.613250] LustreError: Skipped 1 previous similar message [ 7483.360769] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7483.546263] Lustre: server umount lustre-OST0001 complete [ 7486.951889] LustreError: 221792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.979990] LustreError: 221792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 7491.959743] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7492.223792] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7498.087793] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7498.430100] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7529.952489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7535.604851] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7535.632113] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7535.635256] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7535.665163] Lustre: Skipped 1 previous similar message [ 7538.604537] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7547.599792] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7549.748468] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7558.885751] Lustre: server umount lustre-OST0001 complete [ 7578.594089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.613662] Lustre: Skipped 7 previous similar messages [ 7578.617919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7578.628276] Lustre: Skipped 11 previous similar messages [ 7583.580715] Lustre: server umount lustre-OST0000 complete [ 7594.405945] Lustre: server umount lustre-MDT0000 complete [ 7599.029654] LustreError: 218064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781590848 with bad export cookie 17115473919941415845 [ 7599.032777] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7599.045725] LustreError: 218064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7605.574754] Lustre: server umount lustre-MDT0001 complete [ 7632.814504] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:21:21 (1781590881) [ 7640.141765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7640.448586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7640.484804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7645.545309] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7655.461808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7655.663262] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7655.707259] Lustre: 226724:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7661.368922] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7669.082736] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7675.871288] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7683.047411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7683.307066] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7690.666969] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7699.455221] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7710.256637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7710.398418] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7718.328996] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7728.899567] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7731.095240] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7736.348807] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7736.607022] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7742.037194] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7742.243920] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7748.044984] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94c590abe800.ost_server_uuid 50 [ 7749.966772] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff94c590abe800.ost_server_uuid in IDLE state after 0 sec [ 7751.331280] Lustre: Permanently deactivating lustre-OST0001 [ 7751.350208] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7755.565635] Lustre: Permanently reactivating lustre-OST0001 [ 7755.595840] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7755.603381] Lustre: Skipped 2 previous similar messages [ 7760.874366] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7760.880304] LustreError: Skipped 6 previous similar messages [ 7766.133375] Lustre: server umount lustre-OST0001 complete [ 7776.673415] Lustre: server umount lustre-OST0000 complete [ 7796.192366] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7796.457100] Lustre: server umount lustre-MDT0000 complete [ 7800.618067] LustreError: 229601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591050 with bad export cookie 17115473919941417133 [ 7800.623495] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7800.633644] LustreError: 229601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7801.279787] Lustre: server umount lustre-MDT0001 complete [ 7828.081891] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:24:36 (1781591076) [ 7837.178615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.631093] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7837.711629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7837.726641] Lustre: Skipped 2 previous similar messages [ 7838.012128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.020627] Lustre: Skipped 9 previous similar messages [ 7843.367814] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7853.931585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.099793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7854.163613] Lustre: 233423:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7859.776363] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7868.657914] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7875.664993] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7883.981586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7884.271859] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7884.321608] Lustre: Permanently deactivating lustre-OST0000 [ 7884.346341] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7891.853911] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7893.018170] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7900.239253] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7907.080266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.232996] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7914.539746] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 7917.337112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7923.593594] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7925.630632] Lustre: Permanently reactivating lustre-OST0000 [ 7925.650641] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7925.657417] Lustre: Skipped 2 previous similar messages [ 7926.145924] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 33 seconds. I think it's dead, and I am evicting it. exp 000000001059b52d, cur 1781591175 deadline 1781591172 last 1781591142 [ 7926.146061] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7926.169829] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7926.196885] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7926.288700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7948.148900] Lustre: server umount lustre-OST0001 complete [ 7958.642635] Lustre: server umount lustre-OST0000 complete [ 7969.417421] Lustre: server umount lustre-MDT0000 complete [ 7973.354806] LustreError: 233414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591223 with bad export cookie 17115473919941418792 [ 7973.363155] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7973.368489] LustreError: 233414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7973.758592] Lustre: server umount lustre-MDT0001 complete [ 7986.082733] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 7989.568100] Key type lgssc unregistered [ 7989.948869] LNet: 238308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7989.959300] LNetError: 238308:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7989.976802] LNet: Removed LNI 192.168.206.111@tcp [ 7990.889237] Key type .llcrypt unregistered [ 7990.898191] Key type ._llcrypt unregistered [ 8016.272514] Key type ._llcrypt registered [ 8016.277036] Key type .llcrypt registered [ 8016.376086] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 8017.724359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8017.790528] alg: No test for adler32 (adler32-zlib) [ 8018.941940] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 8019.214912] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 8020.888641] Key type lgssc registered [ 8022.093883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8030.919611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8030.947948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8032.479933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8037.031996] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8042.982970] LustreError: 239775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8042.998738] LustreError: 239775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8046.103840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8046.200454] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8046.248202] Lustre: 239762:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8046.268630] Lustre: Permanently deactivating lustre-MDT0001 [ 8046.295824] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8046.601786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8048.616622] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8048.628545] LustreError: 239805:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 8048.645992] LustreError: 239805:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 8048.657153] LustreError: 239805:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 8051.371669] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8058.712202] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8064.397892] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8072.448201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8072.968964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8078.124165] LustreError: 241836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8078.149416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 8081.540418] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8083.427668] LustreError: 241836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8088.582232] LustreError: 241616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8090.940628] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8091.673863] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 8096.743542] LustreError: 241616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8096.764839] LustreError: 241616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8098.210344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8098.451836] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8103.945919] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 8103.948906] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8105.679625] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8114.551130] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8117.749172] Lustre: Permanently reactivating lustre-MDT0001 [ 8117.761713] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8117.765807] Lustre: Skipped 2 previous similar messages [ 8124.221413] Lustre: lustre-MDT0001: Client 35fa1d11-ff0b-47e7-9b94-dc3f23b005d8 (at 192.168.206.11@tcp) reconnecting [ 8130.729251] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8132.817903] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8139.720683] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8139.961798] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8145.815549] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8146.170284] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8149.211615] Lustre: Permanently deactivating lustre-MDT0001 [ 8149.228917] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8149.234328] Lustre: Skipped 2 previous similar messages [ 8155.752338] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8164.322181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8164.334683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.350240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8165.353680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8165.355399] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.365123] Lustre: Skipped 1 previous similar message [ 8165.401687] Lustre: Skipped 1 previous similar message [ 8165.755226] Lustre: server umount lustre-MDT0000 complete [ 8170.123057] LustreError: 239756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591419 with bad export cookie 13786017548661005380 [ 8170.125705] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8170.134203] LustreError: 239756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8170.466383] LustreError: 239769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8170.487344] LustreError: 239769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8170.594958] Lustre: server umount lustre-MDT0001 complete [ 8177.079343] Lustre: server umount lustre-OST0000 complete [ 8189.517122] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:30:37 (1781591437) [ 8201.629129] Lustre: server umount lustre-OST0001 complete [ 8209.772954] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 8219.761497] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 8263.245331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.676804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8263.704219] Lustre: Skipped 2 previous similar messages [ 8263.774727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8263.918972] Lustre: lustre-MDT0000: new disk, initializing [ 8264.046596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8264.065260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8269.042375] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8279.620608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.730475] Lustre: 248035:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8279.819518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8279.827768] Lustre: Skipped 1 previous similar message [ 8279.908712] Lustre: lustre-MDT0001: new disk, initializing [ 8279.999833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8280.039196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8280.056023] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8285.247267] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8295.823692] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8301.459587] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8308.398708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.583513] Lustre: lustre-OST0000: new disk, initializing [ 8308.592310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8308.599639] Lustre: 250004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8308.704124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8314.726513] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8315.471136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8315.484689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8315.579645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8324.313761] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8326.609490] LustreError: 248043:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8336.118733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8336.335995] Lustre: lustre-OST0001: new disk, initializing [ 8336.340740] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8336.346968] Lustre: 251121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8336.688148] LustreError: 248043:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8336.725762] LustreError: 248042:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8343.507569] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8345.702535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8345.720576] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8345.885806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8346.808215] LustreError: 248042:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8353.936855] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8355.746130] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8359.683696] Lustre: lustre-OST0001: Not available for connect from 192.168.206.11@tcp (stopping) [ 8359.691737] Lustre: Skipped 2 previous similar messages [ 8361.444686] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8361.453154] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.463052] LustreError: Skipped 1 previous similar message [ 8361.488587] Lustre: Skipped 2 previous similar messages [ 8364.768491] Lustre: lustre-OST0001: Not available for connect from 192.168.206.11@tcp (stopping) [ 8364.774763] Lustre: Skipped 2 previous similar messages [ 8369.899316] Lustre: lustre-OST0001: Not available for connect from 192.168.206.11@tcp (stopping) [ 8369.921724] Lustre: Skipped 2 previous similar messages [ 8372.194333] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8372.353411] Lustre: server umount lustre-OST0001 complete [ 8375.016887] LustreError: 249987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8397.285879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8397.290609] Lustre: Skipped 1 previous similar message [ 8397.294214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8397.296699] Lustre: Skipped 2 previous similar messages [ 8400.300629] Lustre: server umount lustre-OST0000 complete [ 8407.525686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8407.563057] Lustre: Skipped 1 previous similar message [ 8412.587141] Lustre: server umount lustre-MDT0000 complete [ 8412.643456] LustreError: 248047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8412.660567] LustreError: 248047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8417.929933] LustreError: 248027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591667 with bad export cookie 13786017548661008194 [ 8417.932969] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8417.946649] LustreError: 248027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8424.542731] Lustre: server umount lustre-MDT0001 complete [ 8438.692842] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 8441.888267] Key type lgssc unregistered [ 8442.217966] LNet: 253157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8442.240456] LNetError: 253157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8442.259819] LNet: Removed LNI 192.168.206.111@tcp [ 8443.290720] Key type .llcrypt unregistered [ 8443.292905] Key type ._llcrypt unregistered [ 8487.597317] Key type ._llcrypt registered [ 8487.599606] Key type .llcrypt registered [ 8487.717658] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 8488.957369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8489.087481] alg: No test for adler32 (adler32-zlib) [ 8490.209492] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 8490.454847] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 8492.192232] Key type lgssc registered [ 8493.533894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8502.873407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8502.914483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8504.273007] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8504.322864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8504.561544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8509.493330] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8518.892867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8519.028050] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8519.050960] Lustre: 255366:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8519.511670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8525.267390] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8534.298614] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8540.697486] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8547.823181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8548.094251] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8548.333367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8554.134647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8555.600501] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8563.554064] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8619.670882] Lustre: server umount lustre-OST0000 complete [ 8626.348684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8636.323847] Lustre: 254689:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781591870/real 1781591870] req@00000000f4c273b4 x1868134340409344/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781591886 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8636.354627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8637.408402] Lustre: 254691:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781591870/real 1781591870] req@00000000a989fb32 x1868134340410240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781591886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8637.448578] Lustre: 254691:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8642.001207] Lustre: 254691:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781591875/real 1781591875] req@000000004d810e79 x1868134340411264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781591891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8645.691072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8646.197603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.210504] Lustre: 254688:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781591880/real 1781591880] req@0000000063048059 x1868134340412544/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781591896 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8647.236322] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8647.252206] Lustre: 254688:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8647.302491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8647.313436] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8653.180651] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8660.337894] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8703.461802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.468586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8703.482498] Lustre: Skipped 2 previous similar messages [ 8703.499072] Lustre: Skipped 1 previous similar message [ 8706.465585] Lustre: server umount lustre-OST0000 complete [ 8713.704346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8713.710971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8713.716332] Lustre: Skipped 1 previous similar message [ 8713.733303] Lustre: Skipped 1 previous similar message [ 8717.274483] Lustre: server umount lustre-MDT0000 complete [ 8721.988582] LustreError: 255359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591971 with bad export cookie 9616723266867275450 [ 8721.990515] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8722.008467] LustreError: 255359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8722.383821] Lustre: server umount lustre-MDT0001 complete [ 8735.668474] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 8738.831030] Key type lgssc unregistered [ 8739.165832] LNet: 260789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8739.189387] LNetError: 260789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8739.215229] LNet: Removed LNI 192.168.206.111@tcp [ 8740.127521] Key type .llcrypt unregistered [ 8740.132120] Key type ._llcrypt unregistered [ 8768.357788] Key type ._llcrypt registered [ 8768.361670] Key type .llcrypt registered [ 8768.522534] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 8769.668291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8769.761875] alg: No test for adler32 (adler32-zlib) [ 8771.024822] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 8771.306633] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 8773.024208] Key type lgssc registered [ 8774.292836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8783.387392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8783.426081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.132787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8791.350773] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8794.602122] LustreError: 262212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8794.628033] LustreError: 262212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8799.720587] LustreError: 262211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8802.379301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8802.669830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8809.077162] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8818.899552] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8825.850475] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8835.188211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8835.516446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8843.723407] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8854.325845] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8855.038199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8885.732633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8885.746285] Lustre: Skipped 1 previous similar message [ 8885.746752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8885.749473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8885.766134] Lustre: Skipped 1 previous similar message [ 8890.040976] Lustre: server umount lustre-OST0000 complete [ 8894.951069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8894.965739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.990180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8895.980155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8896.004087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8896.009613] Lustre: Skipped 1 previous similar message [ 8900.497891] Lustre: server umount lustre-MDT0000 complete [ 8901.101310] LustreError: 264474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8901.131735] LustreError: 264474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8905.246908] LustreError: 262192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592154 with bad export cookie 554373602191861342 [ 8905.262798] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8905.264028] LustreError: 262192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8905.908887] Lustre: server umount lustre-MDT0001 complete [ 8921.975789] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 8925.829552] Key type lgssc unregistered [ 8926.313777] LNet: 266521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8926.333528] LNetError: 266521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8926.351813] LNet: Removed LNI 192.168.206.111@tcp [ 8927.509280] Key type .llcrypt unregistered [ 8927.511380] Key type ._llcrypt unregistered [ 8949.635483] Key type ._llcrypt registered [ 8949.637739] Key type .llcrypt registered [ 8949.799374] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 8951.339955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8951.396090] alg: No test for adler32 (adler32-zlib) [ 8952.559538] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 8952.935332] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 8954.640695] Key type lgssc registered [ 8956.174424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8965.761700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8965.773163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.275649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8972.226759] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8976.873931] LustreError: 267743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.907340] LustreError: 267743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8981.386880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8981.820272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8986.217947] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 8993.544642] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8999.964954] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9007.345151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9007.665290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9011.775252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9014.394447] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9022.870345] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9042.407583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9042.427437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9042.440534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9042.465875] Lustre: Skipped 1 previous similar message [ 9042.494289] Lustre: Skipped 1 previous similar message [ 9044.523179] Lustre: server umount lustre-OST0000 complete [ 9052.642599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9052.655911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9052.658059] Lustre: Skipped 1 previous similar message [ 9057.765549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.768611] Lustre: Skipped 1 previous similar message [ 9062.897343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9062.910851] Lustre: Skipped 1 previous similar message [ 9063.392187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9063.590376] Lustre: server umount lustre-MDT0000 complete [ 9068.002882] LustreError: 267738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9068.030400] LustreError: 267738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9069.283487] LustreError: 267726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592318 with bad export cookie 13325738801672226846 [ 9069.296546] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9069.316802] LustreError: 267726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9069.779203] Lustre: server umount lustre-MDT0001 complete [ 9084.148970] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 9087.603320] Key type lgssc unregistered [ 9087.942602] LNet: 271424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9087.952336] LNetError: 271424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9087.970308] LNet: Removed LNI 192.168.206.111@tcp [ 9088.767588] Key type .llcrypt unregistered [ 9088.770280] Key type ._llcrypt unregistered [ 9117.955851] Key type ._llcrypt registered [ 9117.958787] Key type .llcrypt registered [ 9118.117918] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 9119.324602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9119.384688] alg: No test for adler32 (adler32-zlib) [ 9120.634519] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 9121.019554] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 9122.746343] Key type lgssc registered [ 9124.591952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9135.045089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9135.094321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.829245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.323796] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9146.341371] LustreError: 272846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9146.350262] LustreError: 272846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9151.469878] LustreError: 272847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9153.420666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9153.925545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9158.975599] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9167.602496] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9173.982720] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9181.398912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9181.777978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9186.837117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9189.100961] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9197.332458] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9229.809818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9229.811203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9229.840485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9234.251752] Lustre: server umount lustre-OST0000 complete [ 9240.036581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9240.053123] Lustre: Skipped 1 previous similar message [ 9240.058797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9240.066434] Lustre: Skipped 2 previous similar messages [ 9244.433040] Lustre: server umount lustre-MDT0000 complete [ 9245.157162] LustreError: 273574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9245.195658] LustreError: 273574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9249.567320] LustreError: 272826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592499 with bad export cookie 7093412783866605406 [ 9249.575385] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9250.115411] Lustre: server umount lustre-MDT0001 complete [ 9263.435281] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 9266.920690] Key type lgssc unregistered [ 9267.252804] LNet: 277153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9267.265821] LNetError: 277153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9267.281969] LNet: Removed LNI 192.168.206.111@tcp [ 9268.305389] Key type .llcrypt unregistered [ 9268.307686] Key type ._llcrypt unregistered [ 9290.224910] Key type ._llcrypt registered [ 9290.227650] Key type .llcrypt registered [ 9290.351638] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 9292.110880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9292.170486] alg: No test for adler32 (adler32-zlib) [ 9293.273366] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 9293.703035] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 9295.489792] Key type lgssc registered [ 9297.024786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9305.909818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9305.945837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.638950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9312.540162] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9317.862161] LustreError: 278390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9317.883523] LustreError: 278390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9322.201897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9322.640098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9327.655939] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9336.323822] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9342.534790] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9350.545564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9350.986190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9353.015817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9358.013882] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9367.222894] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9382.883073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9382.903487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9382.907451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9382.927249] Lustre: Skipped 1 previous similar message [ 9382.949453] Lustre: Skipped 1 previous similar message [ 9385.554577] Lustre: server umount lustre-OST0000 complete [ 9393.130684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9393.134339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9393.167256] Lustre: Skipped 1 previous similar message [ 9393.205584] Lustre: Skipped 1 previous similar message [ 9395.988598] Lustre: server umount lustre-MDT0000 complete [ 9398.243755] LustreError: 278395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9398.255486] LustreError: 278395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9399.819324] LustreError: 278356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592649 with bad export cookie 8136835824369081711 [ 9399.824620] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9399.829073] LustreError: 278356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9400.328978] Lustre: server umount lustre-MDT0001 complete [ 9412.268359] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 9415.847308] Key type lgssc unregistered [ 9416.194708] LNet: 282092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9416.208194] LNetError: 282092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9416.256194] LNet: Removed LNI 192.168.206.111@tcp [ 9417.341648] Key type .llcrypt unregistered [ 9417.347274] Key type ._llcrypt unregistered [ 9431.322706] Key type ._llcrypt registered [ 9431.326399] Key type .llcrypt registered [ 9431.469765] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9451.498503] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 9468.367730] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 9470.059177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9470.125712] alg: No test for adler32 (adler32-zlib) [ 9471.301342] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 9471.564968] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 9473.298723] Key type lgssc registered [ 9475.037779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9524.289489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9524.356793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9525.805922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9525.882255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9526.001372] Lustre: lustre-MDT0000: new disk, initializing [ 9526.127294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9526.160672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9531.946133] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9543.892491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.036757] Lustre: 285657:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9544.110839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9544.114751] Lustre: Skipped 1 previous similar message [ 9544.175365] Lustre: lustre-MDT0001: new disk, initializing [ 9544.342564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9544.399200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9544.406722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9549.266371] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9559.796273] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9565.562575] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9572.654153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9572.896187] Lustre: lustre-OST0000: new disk, initializing [ 9572.902133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9572.914959] Lustre: 287628:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9573.051564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9580.281673] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9582.207412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9582.230520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9582.380225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9592.302235] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9599.280972] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9599.686491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9604.932606] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9605.162994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9608.167559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9608.195965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9608.198694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9608.228153] Lustre: Skipped 1 previous similar message [ 9613.282423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9613.287458] Lustre: Skipped 2 previous similar messages [ 9613.458155] Lustre: server umount lustre-OST0000 complete [ 9618.409898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9618.424684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9618.439442] Lustre: Skipped 1 previous similar message [ 9623.522751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9623.535293] Lustre: Skipped 1 previous similar message [ 9624.052811] Lustre: server umount lustre-MDT0000 complete [ 9628.597296] LustreError: 285652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592878 with bad export cookie 17024575812451528878 [ 9628.607693] LustreError: 285652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9628.611403] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9629.241033] Lustre: server umount lustre-MDT0001 complete [ 9639.346849] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:54:48 (1781592888) [ 9647.038156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9647.733779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9652.713744] LustreError: 290040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9652.729781] LustreError: 290040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9653.238060] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9657.830126] LustreError: 290041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9662.894506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9662.946687] LustreError: 290040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9663.256311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9668.320396] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9676.280272] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9682.794614] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9689.328304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9689.746363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9697.312900] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9706.003084] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9723.889261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9723.896966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9723.905449] LustreError: Skipped 1 previous similar message [ 9723.945465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9723.947272] Lustre: Skipped 2 previous similar messages [ 9723.966474] Lustre: Skipped 1 previous similar message [ 9726.349915] Lustre: server umount lustre-OST0000 complete [ 9732.070356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.087517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9732.097632] Lustre: Skipped 1 previous similar message [ 9738.415319] Lustre: server umount lustre-MDT0000 complete [ 9739.236909] LustreError: 290041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.263239] LustreError: 290041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9743.330900] LustreError: 290040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9743.355703] LustreError: 290040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9743.568838] LustreError: 292846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592993 with bad export cookie 17024575812451529760 [ 9743.577704] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9743.585496] LustreError: 292846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9744.206442] Lustre: server umount lustre-MDT0001 complete [ 9757.445982] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [ 9761.070357] Key type lgssc unregistered [ 9761.417397] LNet: 293599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9761.424300] LNetError: 293599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9761.461384] LNet: Removed LNI 192.168.206.111@tcp [ 9762.503980] Key type .llcrypt unregistered [ 9762.510073] Key type ._llcrypt unregistered [ 9793.245388] Key type ._llcrypt registered [ 9793.250023] Key type .llcrypt registered [ 9793.423116] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [ 9809.653638] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [ 9810.784456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9810.899652] alg: No test for adler32 (adler32-zlib) [ 9812.101288] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 9812.356662] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [ 9814.072553] Key type lgssc registered [ 9815.225210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9865.777394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9865.832700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9867.181505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9867.236707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9867.388223] Lustre: lustre-MDT0000: new disk, initializing [ 9867.559389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9867.622605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9873.180200] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9885.232053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9885.378194] Lustre: 297432:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9885.426994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9885.432699] Lustre: Skipped 1 previous similar message [ 9885.513774] Lustre: lustre-MDT0001: new disk, initializing [ 9885.692958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9885.773196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9885.789784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9890.899416] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9901.143369] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9907.352863] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9914.894024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9915.154981] Lustre: lustre-OST2710: new disk, initializing [ 9915.162653] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9915.178819] Lustre: 299397:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9915.319988] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9921.965106] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9923.186722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9923.213363] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9923.360443] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9931.572659] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9939.430592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9939.585641] Lustre: lustre-OST03e8: new disk, initializing [ 9939.589327] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9939.594970] Lustre: 300462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9939.676786] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9946.844275] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [ 9949.308238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9949.320261] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9949.427628] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9955.238284] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9964.487663] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9964.844516] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9971.503714] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9971.740850] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9977.058666] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9977.440815] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9982.703907] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9983.048469] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9989.141450] Lustre: server umount lustre-MDT0000 complete [ 9990.627539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9990.638141] Lustre: Skipped 2 previous similar messages [ 9993.311125] LustreError: 297422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593243 with bad export cookie 17845326375515501484 [ 9993.317979] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9993.336550] LustreError: 297422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9993.701527] Lustre: server umount lustre-MDT0001 complete [10007.601544] Lustre: server umount lustre-OST2710 complete [10011.616431] Lustre: 295230:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781593245/real 1781593245] req@000000007beab657 x1868135726650880/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781593261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10011.656502] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10011.680918] Lustre: Skipped 1 previous similar message [10012.956720] Lustre: server umount lustre-OST03e8 complete [10034.244598] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [10044.490950] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [10084.867098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10085.317771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10085.345647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10085.495269] Lustre: lustre-MDT0000: new disk, initializing [10085.571676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10085.588481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10091.737757] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10103.625735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.752830] Lustre: 305726:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10103.806852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10103.810076] Lustre: Skipped 1 previous similar message [10103.887722] Lustre: lustre-MDT0001: new disk, initializing [10103.978688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10104.014565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10104.025131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10108.812360] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10118.140910] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10124.398337] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10131.303743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10131.626340] Lustre: lustre-OST0000: new disk, initializing [10131.636145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10131.644416] Lustre: 307694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10131.750423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10137.964501] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10138.188622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10138.201732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10138.354688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10147.470144] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10152.447837] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10152.710445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10157.847833] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10158.174300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10164.196458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10164.197941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.218783] LustreError: Skipped 1 previous similar message [10164.219505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10164.236478] Lustre: Skipped 1 previous similar message [10166.452154] Lustre: server umount lustre-OST0000 complete [10170.851383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10170.862868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.883504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.888837] Lustre: Skipped 1 previous similar message [10174.446865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10174.461850] Lustre: Skipped 1 previous similar message [10179.553879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10179.566330] Lustre: Skipped 1 previous similar message [10184.674207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10184.679341] Lustre: Skipped 1 previous similar message [10185.184211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10185.424087] Lustre: server umount lustre-MDT0000 complete [10188.937639] LustreError: 305717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593438 with bad export cookie 17845326375515503164 [10188.938800] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10188.947988] LustreError: 305717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10189.349478] Lustre: server umount lustre-MDT0001 complete [10199.522309] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:04:08 (1781593448) [10202.000208] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10204.301726] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:04:12 (1781593452) [10209.214810] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [10238.778174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10238.994688] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10239.024595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10239.210544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10244.395643] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10254.403806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10254.540278] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10254.581644] Lustre: 311554:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10259.350312] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10281.830594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10282.045333] Lustre: Denying initial registration attempt from nid 192.168.206.111@tcp, specified as failover [10282.054235] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10282.071762] LustreError: 313008:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [10282.081469] LustreError: 313008:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [10282.090228] LustreError: 313008:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10282.182897] Lustre: server umount lustre-OST0000 complete [10282.194077] LustreError: 313008:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [10289.635026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10289.649380] Lustre: Skipped 1 previous similar message [10289.662570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10289.669101] Lustre: Skipped 1 previous similar message [10295.623594] Lustre: server umount lustre-MDT0000 complete [10299.873993] LustreError: 313009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10299.905416] LustreError: 313009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10300.683761] LustreError: 311546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593550 with bad export cookie 17845326375515504046 [10300.704755] LustreError: 311546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10300.705269] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10301.217695] Lustre: server umount lustre-MDT0001 complete [10315.409639] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [10319.333323] Key type lgssc unregistered [10319.653635] LNet: 314129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10319.664073] LNetError: 314129:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10319.680553] LNet: Removed LNI 192.168.206.111@tcp [10320.833933] Key type .llcrypt unregistered [10320.837614] Key type ._llcrypt unregistered [10335.041384] Key type ._llcrypt registered [10335.046315] Key type .llcrypt registered [10335.203115] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [10336.442334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10336.609214] alg: No test for adler32 (adler32-zlib) [10337.818879] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [10338.159449] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [10339.896208] Key type lgssc registered [10341.266996] Lustre: Echo OBD driver; http://www.lustre.org/ [10393.374092] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [10404.059059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10404.078358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10405.402123] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10405.448352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10405.656716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10410.627280] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10421.677064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.935821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10422.004802] Lustre: 316894:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10422.352517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10427.456094] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10435.590267] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10441.316056] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10462.070047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10462.443752] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10462.884268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10471.570545] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10482.174974] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10489.835309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10489.839492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10489.849994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10491.150832] Lustre: server umount lustre-OST0000 complete [10499.054560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10499.070220] LustreError: Skipped 1 previous similar message [10499.095125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10499.125534] Lustre: Skipped 1 previous similar message [10499.137450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10499.157218] Lustre: Skipped 1 previous similar message [10505.197226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10505.210356] Lustre: Skipped 3 previous similar messages [10510.307876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10510.319629] Lustre: Skipped 1 previous similar message [10511.328158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10511.719720] Lustre: server umount lustre-MDT0000 complete [10517.115487] LustreError: 316886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593766 with bad export cookie 10228101238747224789 [10517.146748] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10518.047653] Lustre: server umount lustre-MDT0001 complete [10533.658089] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [10537.496918] Key type lgssc unregistered [10538.008656] LNet: 320474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10538.025844] LNetError: 320474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10538.052177] LNet: Removed LNI 192.168.206.111@tcp [10539.084906] Key type .llcrypt unregistered [10539.088800] Key type ._llcrypt unregistered [10574.557810] Key type ._llcrypt registered [10574.566191] Key type .llcrypt registered [10574.835330] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [10576.613661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10576.658573] alg: No test for adler32 (adler32-zlib) [10577.796503] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [10578.081076] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [10579.840233] Key type lgssc registered [10581.401200] Lustre: Echo OBD driver; http://www.lustre.org/ [10592.008632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10592.040731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10593.726348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10599.514280] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10604.011278] LustreError: 321894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.029778] LustreError: 321894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10609.134150] LustreError: 321893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.424684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10612.089685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10618.364398] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10629.019301] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10636.849460] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10648.971994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10649.533746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10658.147827] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10667.825748] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10681.102260] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.11@tcp (stopping) [10682.058683] Lustre: server umount lustre-MDT0000 complete [10683.877983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10683.887608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.914255] LustreError: 321893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.388974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10684.401024] Lustre: Skipped 1 previous similar message [10686.190882] LustreError: 321888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10686.231975] LustreError: 321888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10687.529638] LustreError: 321874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593937 with bad export cookie 5382365422092971097 [10687.532096] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10687.542050] LustreError: 321874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10689.512978] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10689.540570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10691.295681] LustreError: 324390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.308765] LustreError: 324390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10691.328088] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.11@tcp (stopping) [10694.091729] Lustre: server umount lustre-MDT0001 complete [10702.344309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10713.933623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10732.795576] LustreError: 323729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10732.811644] LustreError: 323729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10732.884348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10733.010730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10740.597809] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10748.422128] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10753.522183] LustreError: 325492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10753.548545] LustreError: 325492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10754.156309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10754.912800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10760.227181] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10762.156231] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10771.321811] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10773.885434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10781.183177] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10783.634921] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10790.882148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10790.888293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10790.894437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10790.906646] Lustre: Skipped 1 previous similar message [10790.922609] Lustre: Skipped 2 previous similar messages [10796.014117] Lustre: server umount lustre-OST0000 complete [10806.244122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10806.244145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10806.257810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10806.308890] Lustre: Skipped 1 previous similar message [10807.838977] Lustre: server umount lustre-MDT0000 complete [10810.406149] LustreError: 325490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10810.436309] LustreError: 325490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10812.739322] LustreError: 321873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781594062 with bad export cookie 5382365422092978244 [10812.755246] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10813.354890] Lustre: server umount lustre-MDT0001 complete [10829.078448] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [10833.576091] Key type lgssc unregistered [10834.064389] LNet: 328172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10834.075189] LNetError: 328172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10834.110962] LNet: Removed LNI 192.168.206.111@tcp [10835.480830] Key type .llcrypt unregistered [10835.487075] Key type ._llcrypt unregistered [10867.727654] Key type ._llcrypt registered [10867.730353] Key type .llcrypt registered [10867.934469] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [10869.936454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10870.038312] alg: No test for adler32 (adler32-zlib) [10871.371729] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [10871.751604] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [10873.568268] Key type lgssc registered [10875.521236] Lustre: Echo OBD driver; http://www.lustre.org/ [10887.453190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10887.469821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10889.132063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10894.737799] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10898.405099] LustreError: 329592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10898.427760] LustreError: 329592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10903.529794] LustreError: 329591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10906.245364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10906.957860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10912.289365] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10922.482504] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10930.404291] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10937.826608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10937.840666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10937.877529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10938.852753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10938.853735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.480253] Lustre: server umount lustre-MDT0000 complete [10943.970428] LustreError: 329592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10944.015155] LustreError: 329592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10948.417649] LustreError: 329572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781594198 with bad export cookie 7683899876753677719 [10948.428170] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10949.015397] Lustre: server umount lustre-MDT0001 complete [10970.105627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.717317] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10970.768176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10971.085618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10977.328207] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [10991.655058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10992.030059] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10992.115835] Lustre: 332183:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10992.495276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10999.714436] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11008.518460] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11015.313493] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11026.292868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11026.613991] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11026.628311] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11026.639254] LustreError: 334068:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [11026.652254] LustreError: 334068:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [11026.661645] LustreError: 334068:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [11026.779726] Lustre: server umount lustre-OST0000 complete [11026.788531] LustreError: 334068:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [11039.610724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11039.966357] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11040.190636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11048.738838] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11050.182056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [11058.321254] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11069.202616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.360253] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11069.526399] Lustre: lustre-OST0001: new disk, initializing [11069.538521] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11069.553448] Lustre: 335611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11069.739475] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11078.000645] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11078.775957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11078.798261] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11078.965093] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11088.559733] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11094.522077] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11094.530309] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11094.537669] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11094.543279] Lustre: Skipped 1 previous similar message [11094.560763] Lustre: Skipped 2 previous similar messages [11099.620435] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11104.741768] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11104.756101] Lustre: Skipped 2 previous similar messages [11107.296235] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11107.558214] Lustre: server umount lustre-OST0001 complete [11109.888088] LustreError: 334537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11109.917043] LustreError: 334537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11114.990383] LustreError: 334536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11115.000606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.034193] LustreError: 334536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11115.074090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11119.184253] Lustre: server umount lustre-OST0000 complete [11125.227500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11125.250448] Lustre: Skipped 2 previous similar messages [11130.339446] LustreError: 333209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.357861] LustreError: 333209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11130.374349] Lustre: server umount lustre-MDT0000 complete [11135.418963] LustreError: 332174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781594385 with bad export cookie 7683899876753678272 [11135.431279] LustreError: 332174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11135.434689] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11135.836380] Lustre: server umount lustre-MDT0001 complete [11151.394226] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [11154.689789] Key type lgssc unregistered [11155.173040] LNet: 337573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11155.197915] LNetError: 337573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11155.233508] LNet: Removed LNI 192.168.206.111@tcp [11156.496583] Key type .llcrypt unregistered [11156.499894] Key type ._llcrypt unregistered [11249.145436] Key type ._llcrypt registered [11249.159217] Key type .llcrypt registered [11249.324028] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [11268.974993] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [11271.000983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11271.119332] alg: No test for adler32 (adler32-zlib) [11272.266516] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [11272.600893] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [11274.338408] Key type lgssc registered [11275.976726] Lustre: Echo OBD driver; http://www.lustre.org/ [11331.381709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11331.407945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11332.919399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11333.028901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11333.227153] Lustre: lustre-MDT0000: new disk, initializing [11333.387923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11333.424554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11340.537902] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11355.976571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.172692] Lustre: 342515:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11356.278508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11356.289418] Lustre: Skipped 1 previous similar message [11356.427801] Lustre: lustre-MDT0001: new disk, initializing [11356.635330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11356.688731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11356.709654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11363.664949] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11379.089590] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11388.122751] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11399.342097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11399.846774] Lustre: lustre-OST0000: new disk, initializing [11399.859204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11399.868044] Lustre: 344488:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11400.057421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11405.935296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11405.959286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11406.146323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11409.920609] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11423.118321] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11430.981703] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11431.471239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11439.415608] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11439.870696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11447.270356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11447.273917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.294029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.353046] Lustre: Skipped 1 previous similar message [11449.857223] Lustre: server umount lustre-OST0000 complete [11457.511275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11457.517544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11457.524364] Lustre: Skipped 1 previous similar message [11457.538675] Lustre: Skipped 2 previous similar messages [11462.631037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11462.646756] Lustre: Skipped 1 previous similar message [11463.168346] Lustre: server umount lustre-MDT0000 complete [11468.809710] LustreError: 344487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781594718 with bad export cookie 8409058402412050975 [11468.824460] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11468.840613] LustreError: 344487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11469.476169] Lustre: server umount lustre-MDT0001 complete [11484.307421] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:25:32 (1781594732) [11495.989660] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:25:44 (1781594744) [11508.621184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.185258] LustreError: 347251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.212858] LustreError: 347251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11509.302280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11514.339270] LustreError: 347252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11515.333109] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11518.446736] LustreError: 347251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.561020] LustreError: 347251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11527.270419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11527.814432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11534.462273] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11544.949943] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11552.319476] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11561.218851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11561.777807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11570.010771] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11580.294486] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11589.771486] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11592.313410] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11594.553961] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11597.009942] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11599.768214] Lustre: Failing over lustre-MDT0000 [11600.129217] Lustre: server umount lustre-MDT0000 complete [11600.866648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11600.872645] LustreError: 347248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.890228] Lustre: Skipped 2 previous similar messages [11600.944167] LustreError: 347248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11611.108275] LustreError: 347248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11611.136887] LustreError: 347248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [11617.313184] Lustre: 340738:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781594850/real 1781594850] req@0000000021980e1e x1868137257586048/t0(0) o400->MGC192.168.206.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781594866 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11617.356649] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11621.933247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11628.002549] LustreError: 340736:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f64b85a9 x1868137257591552/t0(0) o250->MGC192.168.206.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11628.057321] LustreError: 347979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.087012] LustreError: 347979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [11628.487129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11628.560663] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11630.304703] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11633.679189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11633.753855] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11633.872202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11634.838414] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11647.009981] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11649.145921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11654.060479] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11656.175157] Lustre: Failing over lustre-MDT0000 [11656.548710] Lustre: server umount lustre-MDT0000 complete [11659.244223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11659.261611] Lustre: Skipped 2 previous similar messages [11661.053048] LustreError: 348180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11661.080709] LustreError: 348180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11674.593750] Lustre: 340740:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781594908/real 1781594908] req@00000000e18c1d83 x1868137257609344/t0(0) o400->MGC192.168.206.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781594924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11674.639199] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11676.847931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11684.841356] LustreError: 340736:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001106c44c x1868137257614848/t0(0) o250->MGC192.168.206.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11685.206156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11685.299710] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11687.661193] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11690.509314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11690.526206] Lustre: Skipped 2 previous similar messages [11690.630142] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11690.756520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11691.606130] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11705.027424] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11707.638936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11716.083101] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11716.092536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11716.101762] Lustre: Skipped 2 previous similar messages [11716.116043] Lustre: Skipped 1 previous similar message [11721.193685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11721.205086] Lustre: Skipped 3 previous similar messages [11724.288497] Lustre: server umount lustre-MDT0000 complete [11726.310446] LustreError: 347979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11726.332224] LustreError: 347979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [11729.817069] LustreError: 347232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781594979 with bad export cookie 8409058402412053691 [11729.821467] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11729.827803] LustreError: 347232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11730.344781] Lustre: server umount lustre-MDT0001 complete [11746.226608] Lustre: server umount lustre-OST0000 complete [11767.430225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.543866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11774.361839] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11787.048208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11794.384989] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11804.578454] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11811.833965] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11821.821175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11831.939230] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11838.933903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11842.898223] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11853.297396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11853.312982] LustreError: Skipped 1 previous similar message [11853.318433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11853.350463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11854.321404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11855.997522] Lustre: server umount lustre-OST0000 complete [11867.592902] Lustre: server umount lustre-MDT0000 complete [11869.667434] LustreError: 353596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11869.685745] LustreError: 353596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11874.110566] LustreError: 353581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781595123 with bad export cookie 8409058402412054433 [11874.114096] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11874.139702] LustreError: 353581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11874.930585] Lustre: server umount lustre-MDT0001 complete [11891.533343] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [11895.379244] Key type lgssc unregistered [11895.749810] LNet: 357136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11895.758039] LNetError: 357136:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11895.804356] LNet: Removed LNI 192.168.206.111@tcp [11896.878637] Key type .llcrypt unregistered [11896.881845] Key type ._llcrypt unregistered [11935.484947] Key type ._llcrypt registered [11935.487547] Key type .llcrypt registered [11935.633732] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [11936.579098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11936.640542] alg: No test for adler32 (adler32-zlib) [11938.531588] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [11938.888865] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [11940.672308] Key type lgssc registered [11942.463673] Lustre: Echo OBD driver; http://www.lustre.org/ [11956.721700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11956.737588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.988824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11967.125551] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11968.486856] LustreError: 358663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11968.497408] LustreError: 358663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11973.608884] LustreError: 358662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11978.731821] LustreError: 358663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11979.712498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.498221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11986.635129] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [11997.648987] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12006.359498] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12017.749060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12018.578047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12028.870468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12028.900646] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12039.271680] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12054.503894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12054.513639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12054.522092] Lustre: Skipped 2 previous similar messages [12054.538554] Lustre: Skipped 1 previous similar message [12059.622428] LustreError: 358658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.646078] LustreError: 358658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12059.673120] Lustre: server umount lustre-MDT0000 complete [12064.318703] LustreError: 361161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781595314 with bad export cookie 5839213660615767476 [12064.323933] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12064.330044] LustreError: 361161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12064.769197] Lustre: server umount lustre-MDT0001 complete [12079.678977] Lustre: server umount lustre-OST0000 complete [12098.941163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12099.842089] LustreError: 362401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12100.038524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12106.642821] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12110.312796] LustreError: 362400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12110.342898] LustreError: 362400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12117.203965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12117.651978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12123.164864] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12133.308582] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12141.148930] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12151.794307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12152.470472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12162.087739] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12170.909580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [12172.939129] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12189.157059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12189.169357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12189.211506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12189.225208] Lustre: Skipped 1 previous similar message [12191.721699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12191.728836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12191.733578] Lustre: Skipped 1 previous similar message [12191.760363] Lustre: Skipped 2 previous similar messages [12194.430446] Lustre: server umount lustre-MDT0000 complete [12196.838496] LustreError: 362397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12196.867524] LustreError: 362397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12199.568185] LustreError: 363135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781595449 with bad export cookie 5839213660615768589 [12199.569403] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12199.587052] LustreError: 363135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12200.126470] Lustre: server umount lustre-MDT0001 complete [12216.602261] Lustre: server umount lustre-OST0000 complete [12241.287026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12241.969801] LustreError: 366316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12242.167451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12249.215555] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12262.376408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12270.020624] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12281.535438] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12290.390339] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12300.196892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12300.796674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12300.806291] Lustre: Skipped 1 previous similar message [12309.152660] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12312.086336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [12320.360872] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12332.527864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12332.541277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12332.542264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12332.554514] Lustre: Skipped 1 previous similar message [12337.634115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12337.643789] Lustre: Skipped 2 previous similar messages [12343.264207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12343.530300] Lustre: server umount lustre-OST0000 complete [12349.925972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12349.933406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12349.953208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12349.975642] Lustre: Skipped 2 previous similar messages [12355.689423] Lustre: server umount lustre-MDT0000 complete [12356.069086] LustreError: 366316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12356.091413] LustreError: 366316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [12361.373811] LustreError: 369103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781595611 with bad export cookie 5839213660615769618 [12361.374917] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12361.388286] LustreError: 369103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12361.880365] Lustre: server umount lustre-MDT0001 complete [12378.069298] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [12382.526229] Key type lgssc unregistered [12382.898486] LNet: 369856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12382.915158] LNetError: 369856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12382.939499] LNet: Removed LNI 192.168.206.111@tcp [12384.306861] Key type .llcrypt unregistered [12384.310627] Key type ._llcrypt unregistered [12420.799669] Key type ._llcrypt registered [12420.805823] Key type .llcrypt registered [12420.947125] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [12422.276128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12422.362603] alg: No test for adler32 (adler32-zlib) [12423.659404] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [12423.883116] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [12425.560234] Key type lgssc registered [12427.288596] Lustre: Echo OBD driver; http://www.lustre.org/ [12436.683766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12436.705706] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12436.714723] LustreError: 371298:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12436.737469] LustreError: 371298:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12436.750435] LustreError: 371298:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12436.760345] LustreError: 371298:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12436.769324] LustreError: 371298:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [12446.367160] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12446.378211] LustreError: 371696:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12446.419913] LustreError: 371696:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12446.443242] LustreError: 371696:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12446.452815] LustreError: 371696:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12446.465902] LustreError: 371696:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [12471.563657] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [12475.566073] Key type lgssc unregistered [12476.035984] LNet: 372701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12476.047629] LNetError: 372701:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12476.081572] LNet: Removed LNI 192.168.206.111@tcp [12477.209666] Key type .llcrypt unregistered [12477.211823] Key type ._llcrypt unregistered [12503.947361] Key type ._llcrypt registered [12503.950911] Key type .llcrypt registered [12504.114925] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [12527.479341] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [12529.471135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12529.772809] alg: No test for adler32 (adler32-zlib) [12531.303490] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [12531.631895] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [12533.377451] Key type lgssc registered [12535.350769] Lustre: Echo OBD driver; http://www.lustre.org/ [12581.657405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12581.694790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12583.107932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12583.158254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12583.286398] Lustre: lustre-MDT0000: new disk, initializing [12583.404348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12583.442053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12588.951622] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12600.996196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12601.232374] Lustre: 375864:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12601.338883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12601.345350] Lustre: Skipped 1 previous similar message [12601.469802] Lustre: lustre-MDT0001: new disk, initializing [12601.679380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12601.722248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12601.737416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12607.727756] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12619.712661] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12628.336704] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12639.004133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12639.416051] Lustre: lustre-OST0000: new disk, initializing [12639.425041] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12639.433657] Lustre: 377834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12639.551910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12648.125588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12648.148186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12648.367607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12648.546742] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12661.777122] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12668.154143] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12668.426560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12674.083475] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12674.338843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12679.137826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12679.153833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12679.193875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12679.206657] Lustre: Skipped 1 previous similar message [12682.838198] Lustre: server umount lustre-OST0000 complete [12688.865528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12688.889791] LustreError: Skipped 1 previous similar message [12688.893796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12688.907484] Lustre: Skipped 1 previous similar message [12688.916894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12694.502614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12694.516963] Lustre: Skipped 3 previous similar messages [12698.593978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12698.615946] Lustre: Skipped 1 previous similar message [12701.152143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12701.500662] Lustre: server umount lustre-MDT0000 complete [12705.678726] LustreError: 375856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781595955 with bad export cookie 4572782002858774224 [12705.684669] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12705.696026] LustreError: 375856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12706.081310] Lustre: server umount lustre-MDT0001 complete [12715.811851] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12717.901518] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:46:06 (1781595966) [12725.065402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12725.419506] LustreError: 380299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12725.441081] LustreError: 380299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12725.493108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12730.219399] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12730.851070] LustreError: 380300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12735.980092] LustreError: 380299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12740.413978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12740.999993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12746.713752] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12756.036845] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12762.643057] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12771.369084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12771.742646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12779.945434] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12789.148930] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12796.601772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12796.805536] Lustre: lustre-OST0001: new disk, initializing [12796.819815] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12796.833874] Lustre: 383137:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12796.978840] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12804.981524] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12805.347159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12805.362980] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12805.616398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12816.038858] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12823.823078] Lustre: lustre-OST0001: Not available for connect from 192.168.206.11@tcp (stopping) [12826.081943] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12826.093784] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12826.115913] Lustre: Skipped 2 previous similar messages [12827.553888] Lustre: server umount lustre-OST0001 complete [12828.895081] LustreError: 382123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12836.333178] LustreError: 383102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12836.355767] LustreError: 383102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12846.569042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12846.571476] LustreError: 382124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12846.573255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12846.573262] Lustre: Skipped 2 previous similar messages [12846.608607] Lustre: Skipped 1 previous similar message [12846.664779] LustreError: 382124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12852.010916] Lustre: server umount lustre-OST0000 complete [12856.802950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.829721] Lustre: Skipped 1 previous similar message [12863.043971] Lustre: server umount lustre-MDT0000 complete [12867.051039] LustreError: 380296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12867.074841] LustreError: 380296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12867.784826] LustreError: 380281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781596117 with bad export cookie 4572782002858775099 [12867.796892] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12867.817859] LustreError: 380281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12874.336083] Lustre: server umount lustre-MDT0001 complete [12887.540531] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [12891.085476] Key type lgssc unregistered [12891.460572] LNet: 385131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12891.484505] LNetError: 385131:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12891.522648] LNet: Removed LNI 192.168.206.111@tcp [12892.404646] Key type .llcrypt unregistered [12892.408835] Key type ._llcrypt unregistered [12958.537794] Key type ._llcrypt registered [12958.543476] Key type .llcrypt registered [12958.723571] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [12960.573469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12960.616328] alg: No test for adler32 (adler32-zlib) [12961.980204] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [12962.313177] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [12964.009958] Key type lgssc registered [12965.881507] Lustre: Echo OBD driver; http://www.lustre.org/ [12977.603695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12977.693957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12979.260786] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12979.300365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12979.571983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12984.637544] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [12995.111723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12995.302277] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12995.329270] Lustre: 387945:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12995.609500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13000.868307] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13009.381981] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13015.829376] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13021.153575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13021.162291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13021.188531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13022.189117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13022.195353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13024.420696] Lustre: server umount lustre-MDT0000 complete [13028.861490] LustreError: 387938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781596278 with bad export cookie 5444668911542571954 [13028.870782] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13028.878109] LustreError: 387938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13029.211631] Lustre: server umount lustre-MDT0001 complete [13041.893121] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:51:30 (1781596290) [13052.282172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13052.811201] LustreError: 390456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13052.823994] LustreError: 390456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13052.888084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13058.021838] LustreError: 390457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13059.080522] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13063.143533] LustreError: 390456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13068.260831] LustreError: 390457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13069.208793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13069.614633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13074.514410] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13082.956546] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13089.662526] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13098.096944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13098.354051] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13098.613331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13106.598896] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13115.273628] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13124.556285] Lustre: Permanently deactivating lustre-OST0000 [13124.597382] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [13133.403826] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13135.624118] LustreError: 393443:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [13135.634692] LustreError: 393443:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [13135.641970] LustreError: 393443:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [13139.910220] LustreError: 393491:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [13139.920037] LustreError: 393491:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [13144.482368] Lustre: server umount lustre-OST0000 complete [13147.790313] LustreError: 393741:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [13147.809793] LustreError: 393741:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [13151.713823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13151.729429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13151.756105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13151.766335] Lustre: Skipped 1 previous similar message [13153.762135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13153.769803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13158.883371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13158.891594] Lustre: Skipped 2 previous similar messages [13160.105783] Lustre: server umount lustre-MDT0000 complete [13164.011549] LustreError: 390457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13164.857662] LustreError: 390439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781596414 with bad export cookie 5444668911542572535 [13164.865499] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13164.874607] LustreError: 390439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13175.467857] Lustre: server umount lustre-MDT0001 complete [13185.330872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13190.451766] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13193.619698] LustreError: 394901:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [13201.207969] Lustre: 395240:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.111@tcp [13203.587787] Lustre: server umount MGS complete [13217.714300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13218.432753] LustreError: 395964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13218.456494] LustreError: 395964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13218.471578] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13218.471593] Lustre: Skipped 1 previous similar message [13218.581345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13224.183552] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13235.023833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13235.446691] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13235.581193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13240.404564] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13250.936961] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13257.849090] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13260.287607] Lustre: Permanently reactivating lustre-OST0000 [13260.330928] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [13260.339227] Lustre: Skipped 2 previous similar messages [13288.429687] LustreError: 397740:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [13298.537490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13299.129688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13307.107885] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13316.036512] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13324.776039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13324.786087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13324.790379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13324.794438] LustreError: Skipped 1 previous similar message [13324.852899] Lustre: Skipped 1 previous similar message [13337.056540] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13337.208552] Lustre: server umount lustre-OST0000 complete [13343.202551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13343.211914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13343.225407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13343.236457] Lustre: Skipped 5 previous similar messages [13348.061316] Lustre: server umount lustre-MDT0000 complete [13350.374362] LustreError: 397815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13350.396498] LustreError: 397815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13351.902289] LustreError: 395944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781596601 with bad export cookie 5444668911542573585 [13351.907052] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13351.913590] LustreError: 395944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13352.387121] Lustre: server umount lustre-MDT0001 complete [13363.873222] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [13367.116889] Key type lgssc unregistered [13367.416446] LNet: 400334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13367.424158] LNetError: 400334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13367.441042] LNet: Removed LNI 192.168.206.111@tcp [13368.378213] Key type .llcrypt unregistered [13368.381367] Key type ._llcrypt unregistered [13391.361895] Key type ._llcrypt registered [13391.365221] Key type .llcrypt registered [13391.507951] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [13410.282650] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [13411.895571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13411.943310] alg: No test for adler32 (adler32-zlib) [13413.284535] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [13413.715944] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [13415.560586] Key type lgssc registered [13417.040966] Lustre: Echo OBD driver; http://www.lustre.org/ [13459.893368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13459.943445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13461.453122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13461.511928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13461.613692] Lustre: lustre-MDT0000: new disk, initializing [13461.790619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13461.831920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13467.489406] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13481.228040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13481.451169] Lustre: 403494:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13481.505823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13481.509872] Lustre: Skipped 1 previous similar message [13481.600487] Lustre: lustre-MDT0001: new disk, initializing [13481.744258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13481.798763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13481.810668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13487.825728] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13499.974761] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13506.601540] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13515.080744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13515.439752] Lustre: lustre-OST0000: new disk, initializing [13515.444282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13515.454162] Lustre: 405465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13515.597877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13523.173982] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13523.603081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13523.614260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13523.770971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13533.428433] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13539.473679] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13539.767416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13545.200965] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13545.418732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13549.550635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13549.550715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13549.557461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13549.567527] LustreError: Skipped 1 previous similar message [13549.594225] Lustre: Skipped 1 previous similar message [13553.799926] Lustre: server umount lustre-OST0000 complete [13558.754861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13558.767107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13558.783269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13558.789664] Lustre: Skipped 1 previous similar message [13559.778640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13559.786906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13564.171326] Lustre: server umount lustre-MDT0000 complete [13568.683837] LustreError: 404299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781596818 with bad export cookie 17375822674058485050 [13568.687760] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13568.699463] LustreError: 404299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13569.046960] Lustre: server umount lustre-MDT0001 complete [13578.291943] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13580.610395] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:00:29 (1781596829) [13589.125775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13589.805852] LustreError: 407927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13589.831026] LustreError: 407927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13589.898840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13595.115609] LustreError: 407928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13595.673652] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13600.234811] LustreError: 407927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13605.347476] LustreError: 407928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13606.033846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13606.591083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13613.415703] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13623.160701] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13629.381619] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13636.507393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13637.001522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13644.201906] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13652.983924] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13655.216981] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13662.180073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13667.811693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13667.818291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13667.835516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13667.841747] Lustre: Skipped 1 previous similar message [13669.859966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13673.474447] Lustre: server umount lustre-OST0000 complete [13678.056689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13678.066329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13678.081633] Lustre: Skipped 1 previous similar message [13678.088817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13678.096681] Lustre: Skipped 2 previous similar messages [13683.929918] Lustre: server umount lustre-MDT0000 complete [13685.217666] LustreError: 407927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13685.241132] LustreError: 407927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13688.019578] LustreError: 407908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781596937 with bad export cookie 17375822674058485925 [13688.042126] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13688.381523] Lustre: server umount lustre-MDT0001 complete [13701.486448] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [13705.297774] Key type lgssc unregistered [13705.739533] LNet: 411667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13705.748053] LNetError: 411667:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13705.765466] LNet: Removed LNI 192.168.206.111@tcp [13706.784672] Key type .llcrypt unregistered [13706.786597] Key type ._llcrypt unregistered [13736.387699] Key type ._llcrypt registered [13736.390191] Key type .llcrypt registered [13736.541126] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [13736.869945] Key type .llcrypt unregistered [13736.877327] Key type ._llcrypt unregistered [13757.436942] Key type ._llcrypt registered [13757.441163] Key type .llcrypt registered [13757.598922] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [13758.845976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13759.078858] alg: No test for adler32 (adler32-zlib) [13760.143996] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [13760.355973] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [13762.024264] Key type lgssc registered [13763.045978] Lustre: Echo OBD driver; http://www.lustre.org/ [13771.471527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13771.485896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13773.023078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13777.783883] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13782.499842] LustreError: 413883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13782.535562] LustreError: 413883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13786.498358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13786.744699] LustreError: 413882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13786.759479] LustreError: 413882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13786.771995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13792.229760] LustreError: 413877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13792.243278] LustreError: 413877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13793.342200] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13797.349924] LustreError: 413883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13797.367213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13797.376220] LustreError: 413883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13801.400943] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13802.476223] LustreError: 413882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13802.507464] LustreError: 413882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13809.585973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13810.064560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13815.416548] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13825.506210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13825.507478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13825.532341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13825.534977] Lustre: Skipped 1 previous similar message [13825.545245] Lustre: Skipped 1 previous similar message [13825.772643] Lustre: server umount lustre-OST0000 complete [13830.627137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13830.634040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13830.648758] Lustre: Skipped 1 previous similar message [13830.675081] Lustre: Skipped 1 previous similar message [13835.750516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13835.757046] Lustre: Skipped 1 previous similar message [13836.295332] Lustre: server umount lustre-MDT0000 complete [13840.221797] LustreError: 413864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781597089 with bad export cookie 2862320506760922242 [13840.235602] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13840.238284] LustreError: 413864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13840.605220] Lustre: server umount lustre-MDT0001 complete [13853.190459] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [13856.696308] Key type lgssc unregistered [13857.108706] LNet: 417030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13857.130689] LNetError: 417030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13857.181887] LNet: Removed LNI 192.168.206.111@tcp [13858.294511] Key type .llcrypt unregistered [13858.303923] Key type ._llcrypt unregistered [13888.659730] Key type ._llcrypt registered [13888.662419] Key type .llcrypt registered [13888.784431] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [13889.814524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13889.847782] alg: No test for adler32 (adler32-zlib) [13891.054739] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [13891.447547] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [13893.184215] Key type lgssc registered [13894.477360] Lustre: Echo OBD driver; http://www.lustre.org/ [13904.953656] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13904.979906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13920.609340] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13925.728389] LustreError: 418423:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.111@tcp: failed processing log, type 4: rc = -110 [13952.544260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13960.759628] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13969.406148] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13978.567823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13979.167431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13979.283450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13984.230520] LustreError: 418448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13984.250469] LustreError: 418448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13985.275904] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [13987.310468] LustreError: 419462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13987.341670] LustreError: 419462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13992.426657] LustreError: 419461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13992.457441] LustreError: 419461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13996.962077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13997.461434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14003.055367] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14012.338604] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14019.644559] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14028.266093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14028.268277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14028.270839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14028.311349] Lustre: Skipped 1 previous similar message [14033.384043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14033.396265] Lustre: Skipped 2 previous similar messages [14034.420362] Lustre: server umount lustre-OST0000 complete [14041.709269] Lustre: server umount lustre-MDT0000 complete [14043.626258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14043.642835] LustreError: 419466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14043.667369] LustreError: 419466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14046.218054] LustreError: 420204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781597295 with bad export cookie 4840529779724675234 [14046.224323] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14046.236374] LustreError: 420204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14046.728994] Lustre: server umount lustre-MDT0001 complete [14059.216369] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [14062.824348] Key type lgssc unregistered [14063.287061] LNet: 421996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14063.304931] LNetError: 421996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14063.340621] LNet: Removed LNI 192.168.206.111@tcp [14064.438241] Key type .llcrypt unregistered [14064.440929] Key type ._llcrypt unregistered [14093.414892] Key type ._llcrypt registered [14093.418416] Key type .llcrypt registered [14093.562118] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [14095.158984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14095.203032] alg: No test for adler32 (adler32-zlib) [14096.313400] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [14096.574093] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [14098.283089] Key type lgssc registered [14099.831969] Lustre: Echo OBD driver; http://www.lustre.org/ [14109.838884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14109.876259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14111.502454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14117.210312] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14121.954762] LustreError: 423413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14121.967455] LustreError: 423413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14127.080384] LustreError: 423414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14128.533892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14129.232215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14135.085930] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14144.064839] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14151.075944] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14158.972870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14159.420894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14166.484415] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14174.582585] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14182.966084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14188.002696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14188.015408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14188.022608] Lustre: Skipped 2 previous similar messages [14188.023318] Lustre: Skipped 1 previous similar message [14190.344713] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.11@tcp (stopping) [14193.944885] Lustre: server umount lustre-MDT0000 complete [14195.417435] LustreError: 425905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14196.201819] LustreError: 423393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.11@tcp arrived at 1781597445 with bad export cookie 18268714592068690443 [14198.017830] LustreError: 423392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781597447 with bad export cookie 18268714592068689778 [14198.022263] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14198.026529] LustreError: 423392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14199.265979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14199.280684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14199.298608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14199.310208] Lustre: Skipped 3 previous similar messages [14201.313701] LustreError: 424341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14201.319304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14201.330388] LustreError: 424341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14204.343877] Lustre: server umount lustre-OST0000 complete [14207.459032] LustreError: 424341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14207.495952] LustreError: 424341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [14212.067867] Lustre: server umount lustre-MDT0001 complete [14224.040828] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [14227.474406] Key type lgssc unregistered [14227.786572] LNet: 427124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14227.793715] LNetError: 427124:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14227.816831] LNet: Removed LNI 192.168.206.111@tcp [14228.853166] Key type .llcrypt unregistered [14228.858736] Key type ._llcrypt unregistered [14256.624505] Key type ._llcrypt registered [14256.631686] Key type .llcrypt registered [14256.803598] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [14257.893325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14257.923106] alg: No test for adler32 (adler32-zlib) [14259.096372] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [14259.375249] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [14261.147260] Key type lgssc registered [14262.436346] Lustre: Echo OBD driver; http://www.lustre.org/ [14272.418779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14272.481273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14274.104164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14280.022117] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14284.273080] LustreError: 428543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14284.292366] LustreError: 428543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14289.380885] LustreError: 428542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14290.417891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14290.751809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14295.669727] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14305.040794] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14312.745501] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14319.862551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14320.328549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14326.406526] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14333.217600] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14343.947653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [14352.353393] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14352.364096] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14352.386100] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14354.185356] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.11@tcp (stopping) [14354.404488] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14359.271282] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.11@tcp (stopping) [14359.282855] Lustre: Skipped 2 previous similar messages [14364.392826] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.11@tcp (stopping) [14364.401412] Lustre: Skipped 2 previous similar messages [14366.176173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14366.378768] Lustre: server umount lustre-MDT0001 complete [14369.766822] LustreError: 428543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14369.798057] LustreError: 428543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14374.883878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14374.890904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14374.894517] LustreError: 428542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14374.903871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14374.920730] Lustre: Skipped 2 previous similar messages [14375.714019] Lustre: server umount lustre-OST0000 complete [14380.001547] LustreError: 429271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14380.973093] Lustre: server umount lustre-MDT0000 complete [14399.919854] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [14403.455272] Key type lgssc unregistered [14403.921335] LNet: 432252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14403.935826] LNetError: 432252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14403.958554] LNet: Removed LNI 192.168.206.111@tcp [14405.173468] Key type .llcrypt unregistered [14405.176598] Key type ._llcrypt unregistered [14436.107916] Key type ._llcrypt registered [14436.109901] Key type .llcrypt registered [14436.277648] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [14455.253192] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [14456.580250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14456.610353] alg: No test for adler32 (adler32-zlib) [14457.768618] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [14458.034817] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [14459.696228] Key type lgssc registered [14461.403132] Lustre: Echo OBD driver; http://www.lustre.org/ [14506.583477] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [14528.509574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14528.542593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14529.993769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14530.031861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14530.133403] Lustre: lustre-MDT0000: new disk, initializing [14530.254457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14530.277976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14536.291230] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14553.723432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14554.150977] Lustre: lustre-OST0000: new disk, initializing [14554.157393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14554.160696] Lustre: Skipped 1 previous similar message [14554.167972] Lustre: 437291:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14554.304487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14560.372305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14560.384491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14560.429728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14562.589094] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14573.541402] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14592.348464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14592.558458] Lustre: 436270:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14592.664066] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14592.777458] Lustre: lustre-MDT0001: new disk, initializing [14592.965102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14593.044671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14593.061455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14598.460162] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14609.908476] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14610.352704] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14616.002076] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14616.254742] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14622.492794] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.11@tcp (stopping) [14623.207728] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14623.221840] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14623.231389] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14623.244126] Lustre: Skipped 1 previous similar message [14627.558171] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.11@tcp (stopping) [14627.577866] Lustre: Skipped 1 previous similar message [14628.932776] Lustre: server umount lustre-MDT0001 complete [14638.566637] LustreError: 436277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14638.585915] LustreError: 436277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [14643.683641] LustreError: 436279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14643.715451] LustreError: 436279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14648.805184] LustreError: 436277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14648.874372] LustreError: 436277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14652.386358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14652.395101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14652.405561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14652.409961] Lustre: Skipped 2 previous similar messages [14653.925065] LustreError: 436282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14656.387927] Lustre: server umount lustre-OST0000 complete [14659.043086] LustreError: 436283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14668.841572] Lustre: server umount lustre-MDT0000 complete [14688.163561] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [14692.046840] Key type lgssc unregistered [14692.370167] LNet: 441045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14692.391811] LNetError: 441045:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14692.410853] LNet: Removed LNI 192.168.206.111@tcp [14693.297169] Key type .llcrypt unregistered [14693.300320] Key type ._llcrypt unregistered [14740.840712] Key type ._llcrypt registered [14740.849254] Key type .llcrypt registered [14741.061206] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [14741.579379] Key type .llcrypt unregistered [14741.581268] Key type ._llcrypt unregistered [14763.122406] Key type ._llcrypt registered [14763.137293] Key type .llcrypt registered [14763.391763] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [14765.488976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14765.526179] alg: No test for adler32 (adler32-zlib) [14766.767917] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [14767.096224] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [14768.832225] Key type lgssc registered [14770.366564] Lustre: Echo OBD driver; http://www.lustre.org/ [14799.755720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14799.788223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14801.112860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14801.181709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14801.309960] Lustre: lustre-MDT0000: new disk, initializing [14801.457036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14801.535248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14807.044387] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14818.497390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14818.692644] Lustre: 444424:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14818.762762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14818.767656] Lustre: Skipped 1 previous similar message [14818.884768] Lustre: lustre-MDT0001: new disk, initializing [14819.032376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14819.087891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14819.111445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14824.266870] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14834.965146] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14841.414045] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14849.342122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14849.577697] Lustre: lustre-OST0000: new disk, initializing [14849.582254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14849.592479] Lustre: 446394:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14849.692729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14855.967567] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14857.829478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14857.848955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14857.997265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14865.752073] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14873.572910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14873.581775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14873.585115] Lustre: Skipped 1 previous similar message [14878.699383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14878.705510] Lustre: Skipped 3 previous similar messages [14879.018282] Lustre: server umount lustre-MDT0000 complete [14883.694426] LustreError: 444416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781598133 with bad export cookie 1573753336638692811 [14883.704434] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14883.718870] LustreError: 444416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14883.809453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14883.826112] Lustre: Skipped 2 previous similar messages [14883.841778] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14884.186431] Lustre: server umount lustre-MDT0001 complete [14890.674762] Lustre: server umount lustre-OST0000 complete [14903.169682] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:22:31 (1781598151) [14912.363918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14912.996953] LustreError: 448422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14913.016449] LustreError: 448422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14913.112706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14918.115229] LustreError: 448423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14918.135387] LustreError: 448423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14918.682403] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14923.236329] LustreError: 448422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14928.360136] LustreError: 448423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14928.554290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14929.099628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14934.558341] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14943.503051] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14950.400715] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14971.838240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14972.280547] Lustre: Found index 0 for lustre-OST0000, updating log [14972.613387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14980.549145] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [14983.876147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14989.935646] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14997.900157] Lustre: server umount lustre-OST0000 complete [15014.883399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15014.893979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15014.901705] Lustre: Skipped 1 previous similar message [15014.931067] Lustre: Skipped 1 previous similar message [15015.904256] Lustre: 442958:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781598249/real 1781598249] req@00000000e6690fd8 x1868140921299840/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781598265 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15021.028612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15021.039263] Lustre: Skipped 3 previous similar messages [15021.088361] Lustre: 442955:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781598254/real 1781598254] req@00000000e1f121d4 x1868140921301632/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781598270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15021.136611] Lustre: 442955:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15026.145660] Lustre: 442956:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781598259/real 1781598259] req@00000000f90b173c x1868140921302784/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781598275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15029.216698] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15029.488476] Lustre: server umount lustre-MDT0000 complete [15030.307281] LNet: 1 local NIs in recovery (showing 1): 192.168.206.111@tcp [15031.266139] Lustre: 442958:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781598264/real 1781598264] req@000000000c3d1616 x1868140921304448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781598280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15032.301972] LustreError: 448418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15032.327991] LustreError: 448418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15034.934663] LustreError: 448404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781598284 with bad export cookie 1573753336638694211 [15034.937338] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15034.941959] LustreError: 448404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15035.537930] Lustre: server umount lustre-MDT0001 complete [15046.795787] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:24:55 (1781598295) [15682.535640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15683.358134] LustreError: 454839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15683.545270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15688.677572] LustreError: 454840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15690.639899] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [15693.794919] LustreError: 454839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15699.943042] LustreError: 454840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15699.964381] LustreError: 454840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15702.552109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15703.109743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15708.443663] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [15718.119694] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15724.610061] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15734.880205] LNet: 1 local NIs in recovery (showing 1): 192.168.206.111@tcp [15746.096120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15746.415665] Lustre: Found index 0 for lustre-OST0000, updating log [15746.797236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15755.060273] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [15765.298518] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15768.751478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15768.786609] Lustre: Skipped 1 previous similar message [15768.799805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15768.814543] Lustre: Skipped 4 previous similar messages [15781.857425] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15782.104676] Lustre: server umount lustre-OST0000 complete [15788.580228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15788.586269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15788.600772] Lustre: Skipped 1 previous similar message [15788.615026] Lustre: Skipped 1 previous similar message [15790.694676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15790.711716] Lustre: Skipped 3 previous similar messages [15793.964373] Lustre: server umount lustre-MDT0000 complete [15794.791817] LustreError: 455570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15794.814966] LustreError: 455570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15796.320176] LNet: 1 local NIs in recovery (showing 1): 192.168.206.111@tcp [15798.947576] LustreError: 454822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781599048 with bad export cookie 1573753336638695296 [15798.950549] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15798.959079] LustreError: 454822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15799.372284] Lustre: server umount lustre-MDT0001 complete [15841.762250] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [15857.879857] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [15914.210779] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:39:22 (1781599162) [15938.105829] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [15942.087415] Key type lgssc unregistered [15942.505920] LNet: 461477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15942.540561] LNetError: 461477:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16009.129581] LNet: Removed LNI 192.168.206.111@tcp [16010.431029] Key type .llcrypt unregistered [16010.434174] Key type ._llcrypt unregistered [16033.611253] Key type ._llcrypt registered [16033.613929] Key type .llcrypt registered [16033.746214] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [16035.286498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16035.482820] alg: No test for adler32 (adler32-zlib) [16036.704862] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [16037.019063] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [16038.760333] Key type lgssc registered [16040.267132] Lustre: Echo OBD driver; http://www.lustre.org/ [16045.804389] LNet: Added LNI 192.168.206.111@tcp100 [8/256/0/180] [16046.903599] LNet: Added LNI 192.168.206.111@tcp101 [8/256/0/180] [16048.250902] LNet: Added LNI 192.168.206.111@tcp102 [8/256/0/180] [16049.202717] LNet: Added LNI 192.168.206.111@tcp103 [8/256/0/180] [16050.217943] LNet: Added LNI 192.168.206.111@tcp104 [8/256/0/180] [16051.374567] LNet: Added LNI 192.168.206.111@tcp105 [8/256/0/180] [16052.327921] LNet: Added LNI 192.168.206.111@tcp106 [8/256/0/180] [16053.444465] LNet: Added LNI 192.168.206.111@tcp107 [8/256/0/180] [16054.514385] LNet: Added LNI 192.168.206.111@tcp108 [8/256/0/180] [16055.415816] LNet: Added LNI 192.168.206.111@tcp109 [8/256/0/180] [16056.517600] LNet: Added LNI 192.168.206.111@tcp110 [8/256/0/180] [16057.544545] LNet: Added LNI 192.168.206.111@tcp111 [8/256/0/180] [16058.567507] LNet: Added LNI 192.168.206.111@tcp112 [8/256/0/180] [16059.626903] LNet: Added LNI 192.168.206.111@tcp113 [8/256/0/180] [16060.702913] LNet: Added LNI 192.168.206.111@tcp114 [8/256/0/180] [16061.985298] LNet: Added LNI 192.168.206.111@tcp115 [8/256/0/180] [16062.986664] LNet: Added LNI 192.168.206.111@tcp116 [8/256/0/180] [16064.084148] LNet: Added LNI 192.168.206.111@tcp117 [8/256/0/180] [16065.187576] LNet: Added LNI 192.168.206.111@tcp118 [8/256/0/180] [16066.232885] LNet: Added LNI 192.168.206.111@tcp119 [8/256/0/180] [16067.448111] LNet: Added LNI 192.168.206.111@tcp120 [8/256/0/180] [16068.593339] LNet: Added LNI 192.168.206.111@tcp121 [8/256/0/180] [16069.887226] LNet: Added LNI 192.168.206.111@tcp122 [8/256/0/180] [16071.093302] LNet: Added LNI 192.168.206.111@tcp123 [8/256/0/180] [16072.327551] LNet: Added LNI 192.168.206.111@tcp124 [8/256/0/180] [16073.322120] LNet: Added LNI 192.168.206.111@tcp125 [8/256/0/180] [16074.454283] LNet: Added LNI 192.168.206.111@tcp126 [8/256/0/180] [16075.639447] LNet: Added LNI 192.168.206.111@tcp127 [8/256/0/180] [16076.992866] LNet: Added LNI 192.168.206.111@tcp128 [8/256/0/180] [16078.336293] LNet: Added LNI 192.168.206.111@tcp129 [8/256/0/180] [16079.596400] LNet: Added LNI 192.168.206.111@tcp130 [8/256/0/180] [16080.884912] LNet: Added LNI 192.168.206.111@tcp131 [8/256/0/180] [16082.204785] LNet: Added LNI 192.168.206.111@tcp132 [8/256/0/180] [16083.343459] LNet: Added LNI 192.168.206.111@tcp133 [8/256/0/180] [16084.553364] LNet: Added LNI 192.168.206.111@tcp134 [8/256/0/180] [16085.684350] LNet: Added LNI 192.168.206.111@tcp135 [8/256/0/180] [16094.185795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16094.227780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16095.454528] Lustre: 464505:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.111@tcp: can't add NID 192.168.206.111@tcp130: rc = -75 [16095.572024] LNetError: 461964:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.111 [16095.585331] LNetError: Protocol error connecting to 192.168.206.111@tcp114 at host 192.168.206.111:988: is it running a compatible version of Lustre? [16095.603640] LNetError: Skipped 4 previous similar messages [16095.732743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16095.768828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16095.851632] Lustre: lustre-MDT0000: new disk, initializing [16095.993445] Lustre: 464561:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.111@tcp: can't add NID 192.168.206.111@tcp130: rc = -75 [16096.007347] Lustre: 464561:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [16096.030386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16096.068848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16096.425805] LNet: 8 local NIs in recovery (showing 8): 192.168.206.111@tcp121, 192.168.206.111@tcp122, 192.168.206.111@tcp125, 192.168.206.111@tcp126, 192.168.206.111@tcp124, 192.168.206.111@tcp129, 192.168.206.111@tcp123, 192.168.206.111@tcp128 [16096.435903] LNetError: 461964:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.111 [16096.469837] LNetError: 461964:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [16096.478853] LNetError: Protocol error connecting to 192.168.206.111@tcp101 at host 192.168.206.111:988: is it running a compatible version of Lustre? [16096.494684] LNetError: Skipped 8 previous similar messages [16098.531235] LNetError: 464527:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.111 [16098.548027] LNetError: 464527:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [16098.560542] LNetError: Protocol error connecting to 192.168.206.111@tcp126 at host 192.168.206.111:988: is it running a compatible version of Lustre? [16102.271904] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [16102.628533] LNetError: 464534:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.111 [16102.637866] LNetError: Protocol error connecting to 192.168.206.111@tcp126 at host 192.168.206.111:988: is it running a compatible version of Lustre? [16110.818043] LNetError: 461963:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.111 [16110.839097] LNetError: Protocol error connecting to 192.168.206.111@tcp126 at host 192.168.206.111:988: is it running a compatible version of Lustre? [16116.322067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16116.544936] Lustre: 464522:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16116.616782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16116.619645] Lustre: Skipped 1 previous similar message [16116.734379] Lustre: lustre-MDT0001: new disk, initializing [16116.793820] Lustre: 465324:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.111@tcp: can't add NID 192.168.206.111@tcp130: rc = -75 [16116.802779] Lustre: 465324:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [16116.998582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16117.069039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16117.082476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16122.775375] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [16125.007920] Lustre: 465676:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.111@tcp: can't add NID 192.168.206.111@tcp130: rc = -75 [16125.026560] Lustre: 465676:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [16126.113700] LNetError: 464586:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.111 [16126.126421] LNetError: Protocol error connecting to 192.168.206.111@tcp124 at host 192.168.206.111:988: is it running a compatible version of Lustre? [16133.322347] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16139.880245] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16145.570314] LNetError: 464526:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.111 [16145.583728] LNetError: 464526:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [16145.591334] LNetError: Protocol error connecting to 192.168.206.111@tcp120 at host 192.168.206.111:988: is it running a compatible version of Lustre? [16145.599775] LNetError: Skipped 11 previous similar messages [16147.517967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16147.857791] Lustre: lustre-OST0000: new disk, initializing [16147.861310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16147.868539] Lustre: 466498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16147.971655] Lustre: 466503:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.111@tcp: can't add NID 192.168.206.111@tcp130: rc = -75 [16147.981975] Lustre: 466503:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [16148.057767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16155.589347] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [16156.896392] LNet: 1 local NIs in recovery (showing 1): 192.168.206.111@tcp126 [16157.228168] Lustre: 466924:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.111@tcp: can't add NID 192.168.206.111@tcp130: rc = -75 [16157.243722] Lustre: 466924:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [16157.346451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16157.367125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16157.541554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16166.007188] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16178.146066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16178.149133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16178.168268] Lustre: Skipped 1 previous similar message [16178.175776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16180.389399] LNetError: 461963:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.111 [16180.402389] LNetError: 461963:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [16180.412534] LNetError: Protocol error connecting to 192.168.206.111@tcp101 at host 192.168.206.111:988: is it running a compatible version of Lustre? [16180.422198] LNetError: Skipped 13 previous similar messages [16180.747493] Lustre: server umount lustre-OST0000 complete [16188.391766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16188.401462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16188.411101] Lustre: Skipped 1 previous similar message [16188.444520] Lustre: Skipped 2 previous similar messages [16193.508580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16193.524821] Lustre: Skipped 1 previous similar message [16197.601646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16197.607958] Lustre: Skipped 1 previous similar message [16199.141089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16199.494671] Lustre: server umount lustre-MDT0000 complete [16204.667506] LustreError: 464514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781599454 with bad export cookie 2022314525039481317 [16204.677139] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16204.686882] LustreError: 464514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16205.316584] Lustre: server umount lustre-MDT0001 complete [16219.191325] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [16222.720296] Key type lgssc unregistered [16223.354346] LNet: 468415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16223.370383] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16223.379734] LNet: Removed LNI 192.168.206.111@tcp [16224.417500] LNet: Removed LNI 192.168.206.111@tcp100 [16224.419685] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16224.433324] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [16224.456428] LNet: Removed LNI 192.168.206.111@tcp101 [16225.504187] LNet: Removed LNI 192.168.206.111@tcp102 [16225.513385] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16225.528061] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [16225.546296] LNet: Removed LNI 192.168.206.111@tcp103 [16226.592327] LNet: Removed LNI 192.168.206.111@tcp104 [16227.616627] LNet: Removed LNI 192.168.206.111@tcp105 [16227.626532] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16227.643639] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [16227.666702] LNet: Removed LNI 192.168.206.111@tcp106 [16228.705650] LNet: Removed LNI 192.168.206.111@tcp107 [16229.728668] LNet: Removed LNI 192.168.206.111@tcp108 [16230.752657] LNet: Removed LNI 192.168.206.111@tcp109 [16231.786789] LNet: Removed LNI 192.168.206.111@tcp110 [16231.797310] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16231.814221] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [16231.824120] LNet: Removed LNI 192.168.206.111@tcp111 [16232.865491] LNet: Removed LNI 192.168.206.111@tcp112 [16233.888351] LNet: Removed LNI 192.168.206.111@tcp113 [16234.912198] LNet: Removed LNI 192.168.206.111@tcp114 [16235.936870] LNet: Removed LNI 192.168.206.111@tcp115 [16236.961169] LNet: Removed LNI 192.168.206.111@tcp116 [16237.984923] LNet: Removed LNI 192.168.206.111@tcp117 [16239.011378] LNet: Removed LNI 192.168.206.111@tcp118 [16240.033880] LNet: Removed LNI 192.168.206.111@tcp119 [16240.047392] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16240.059481] LNetError: 468415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [16240.071442] LNet: Removed LNI 192.168.206.111@tcp120 [16241.120547] LNet: Removed LNI 192.168.206.111@tcp121 [16242.147025] LNet: Removed LNI 192.168.206.111@tcp122 [16243.168266] LNet: Removed LNI 192.168.206.111@tcp123 [16244.192264] LNet: Removed LNI 192.168.206.111@tcp124 [16245.216269] LNet: Removed LNI 192.168.206.111@tcp125 [16245.225360] LNet: Removed LNI 192.168.206.111@tcp126 [16246.240587] LNet: Removed LNI 192.168.206.111@tcp127 [16247.264215] LNet: Removed LNI 192.168.206.111@tcp128 [16248.289080] LNet: Removed LNI 192.168.206.111@tcp129 [16248.294209] LNet: Removed LNI 192.168.206.111@tcp130 [16248.299618] LNet: Removed LNI 192.168.206.111@tcp131 [16248.303054] LNet: Removed LNI 192.168.206.111@tcp132 [16248.306162] LNet: Removed LNI 192.168.206.111@tcp133 [16248.310346] LNet: Removed LNI 192.168.206.111@tcp134 [16248.328908] LNet: Removed LNI 192.168.206.111@tcp135 [16249.514061] Key type .llcrypt unregistered [16249.523108] Key type ._llcrypt unregistered [16283.823586] Key type ._llcrypt registered [16283.825876] Key type .llcrypt registered [16283.973487] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [16305.390834] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [16307.348179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16307.380935] alg: No test for adler32 (adler32-zlib) [16308.508833] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [16308.812904] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [16310.611081] Key type lgssc registered [16312.055987] Lustre: Echo OBD driver; http://www.lustre.org/ [16364.026927] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:46:51 (1781599611) [16390.546928] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [16393.996740] Key type lgssc unregistered [16394.558969] LNet: 472512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16394.577120] LNetError: 472512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16394.597733] LNet: Removed LNI 192.168.206.111@tcp [16395.804493] Key type .llcrypt unregistered [16395.810192] Key type ._llcrypt unregistered [16420.634506] Key type ._llcrypt registered [16420.636494] Key type .llcrypt registered [16420.824359] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [16422.655712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16422.686615] alg: No test for adler32 (adler32-zlib) [16423.880182] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [16424.160972] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [16425.960207] Key type lgssc registered [16427.545660] Lustre: Echo OBD driver; http://www.lustre.org/ [16440.097244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16440.169311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16441.651205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16441.777232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16441.923750] Lustre: lustre-MDT0000: new disk, initializing [16442.051502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16442.068854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16447.867441] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [16463.395122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16463.686713] Lustre: 473725:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16463.810845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16463.826979] Lustre: Skipped 1 previous similar message [16464.012510] Lustre: lustre-MDT0001: new disk, initializing [16464.196164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16464.264928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16464.282979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16470.534593] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [16483.401416] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16490.674282] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16499.892020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16500.321498] Lustre: lustre-OST0000: new disk, initializing [16500.328977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16500.350543] Lustre: 475695:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16500.523403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16509.117423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16509.138775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16509.365812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16509.833653] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [16521.465844] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16526.552165] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing lnet_if_list [16534.508671] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16534.564872] LNet: Added LNI 192.168.206.111@tcp10 [8/256/0/180] [16542.711385] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16542.793822] LNet: Added LNI 192.168.206.111@tcp11 [8/256/0/180] [16548.754421] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16548.806566] LNet: Added LNI 192.168.206.111@tcp12 [8/256/0/180] [16554.824712] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16554.857900] LNet: Added LNI 192.168.206.111@tcp13 [8/256/0/180] [16562.192903] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16562.264919] LNet: Added LNI 192.168.206.111@tcp14 [8/256/0/180] [16569.068220] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16569.109716] LNet: Added LNI 192.168.206.111@tcp15 [8/256/0/180] [16576.211390] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16576.275438] LNet: Added LNI 192.168.206.111@tcp16 [8/256/0/180] [16583.892383] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16583.952488] LNet: Added LNI 192.168.206.111@tcp17 [8/256/0/180] [16591.348489] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16591.403341] LNet: Added LNI 192.168.206.111@tcp18 [8/256/0/180] [16598.818098] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16598.886917] LNet: Added LNI 192.168.206.111@tcp19 [8/256/0/180] [16607.626759] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16607.670876] LNet: Added LNI 192.168.206.111@tcp20 [8/256/0/180] [16617.999227] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16618.048910] LNet: Added LNI 192.168.206.111@tcp21 [8/256/0/180] [16626.616395] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16626.702818] LNet: Added LNI 192.168.206.111@tcp22 [8/256/0/180] [16635.063678] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16635.108449] LNet: Added LNI 192.168.206.111@tcp23 [8/256/0/180] [16643.494392] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16643.528256] LNet: Added LNI 192.168.206.111@tcp24 [8/256/0/180] [16652.224534] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16652.293094] LNet: Added LNI 192.168.206.111@tcp25 [8/256/0/180] [16662.103672] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16662.170144] LNet: Added LNI 192.168.206.111@tcp26 [8/256/0/180] [16671.520850] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16671.556106] LNet: Added LNI 192.168.206.111@tcp27 [8/256/0/180] [16681.014852] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16681.049593] LNet: Added LNI 192.168.206.111@tcp28 [8/256/0/180] [16689.565329] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16689.620402] LNet: Added LNI 192.168.206.111@tcp29 [8/256/0/180] [16698.385534] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16698.430239] LNet: Added LNI 192.168.206.111@tcp30 [8/256/0/180] [16707.156384] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16707.235286] LNet: Added LNI 192.168.206.111@tcp31 [8/256/0/180] [16717.786815] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16717.828685] LNet: Added LNI 192.168.206.111@tcp32 [8/256/0/180] [16727.319342] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16727.342929] LNet: Added LNI 192.168.206.111@tcp33 [8/256/0/180] [16737.452151] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16737.520163] LNet: Added LNI 192.168.206.111@tcp34 [8/256/0/180] [16747.403422] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16747.454054] LNet: Added LNI 192.168.206.111@tcp35 [8/256/0/180] [16756.985736] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16757.037830] LNet: Added LNI 192.168.206.111@tcp36 [8/256/0/180] [16766.655795] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16766.685829] LNet: Added LNI 192.168.206.111@tcp37 [8/256/0/180] [16775.042594] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16775.077317] LNet: Added LNI 192.168.206.111@tcp38 [8/256/0/180] [16782.727775] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16782.785968] LNet: Added LNI 192.168.206.111@tcp39 [8/256/0/180] [16791.075446] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16791.139646] LNet: Added LNI 192.168.206.111@tcp40 [8/256/0/180] [16801.717635] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16801.775250] LNet: Added LNI 192.168.206.111@tcp41 [8/256/0/180] [16813.408751] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16813.464821] LNet: Added LNI 192.168.206.111@tcp42 [8/256/0/180] [16823.521308] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16823.564103] LNet: Added LNI 192.168.206.111@tcp43 [8/256/0/180] [16835.227245] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16835.255088] LNet: Added LNI 192.168.206.111@tcp44 [8/256/0/180] [16845.642233] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16845.701054] LNet: Added LNI 192.168.206.111@tcp45 [8/256/0/180] [16857.139942] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16857.185729] LNet: Added LNI 192.168.206.111@tcp46 [8/256/0/180] [16870.344713] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16870.375305] LNet: Added LNI 192.168.206.111@tcp47 [8/256/0/180] [16881.706751] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16881.763642] LNet: Added LNI 192.168.206.111@tcp48 [8/256/0/180] [16893.126783] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16893.161521] LNet: Added LNI 192.168.206.111@tcp49 [8/256/0/180] [16904.343338] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16904.421973] LNet: Added LNI 192.168.206.111@tcp50 [8/256/0/180] [16930.274235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16930.281786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16930.286194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16930.326053] Lustre: Skipped 2 previous similar messages [16934.320362] Lustre: server umount lustre-MDT0000 complete [16939.086269] LustreError: 473719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781600188 with bad export cookie 14473332002828807522 [16939.094536] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16939.099960] LustreError: 473719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16939.588596] Lustre: server umount lustre-MDT0001 complete [16956.898784] Lustre: 472999:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781600190/real 1781600190] req@00000000fdbdb88a x1868142658932736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781600206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16956.937161] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16961.057244] Lustre: 472999:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781600194/real 1781600194] req@00000000ece60277 x1868142658932864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781600210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16963.589722] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [16965.088160] Lustre: 472998:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781600198/real 1781600198] req@000000004c3774a5 x1868142658933248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781600214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16965.136984] Lustre: 472998:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16970.272912] Lustre: 472999:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781600203/real 1781600203] req@0000000021e10cb2 x1868142658933376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781600219 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16976.983725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16983.590439] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc8db9ac608e7782f [16983.623108] Lustre: MGC192.168.206.111@tcp: Connection restored to 0@lo (at 0@lo) [16984.160532] LustreError: 475688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16984.193572] LustreError: 475688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [16984.241719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16986.274334] LustreError: 475689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16986.283541] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16990.865745] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [16991.733870] LustreError: 475688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16996.836576] LustreError: 475689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17001.959252] LustreError: 499958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17001.996609] LustreError: 499958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [17003.548190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17004.196941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17009.650729] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [17010.420394] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17022.217886] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17032.386522] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [17060.325510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17060.334653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17060.353655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17060.364472] Lustre: Skipped 2 previous similar messages [17062.963261] Lustre: server umount lustre-OST0000 complete [17070.577942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17070.582967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17070.593188] Lustre: Skipped 1 previous similar message [17070.598118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17070.612290] Lustre: Skipped 1 previous similar message [17073.848863] Lustre: server umount lustre-MDT0000 complete [17076.207486] LustreError: 500615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17076.249884] LustreError: 500615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [17078.554356] LustreError: 473717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781600328 with bad export cookie 14473332002828810287 [17078.555945] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17078.564349] LustreError: 473717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17079.145468] Lustre: server umount lustre-MDT0001 complete [17093.617742] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [17097.386091] Key type lgssc unregistered [17097.964665] LNet: 502767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17097.981065] LNetError: 502767:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [17097.995105] LNet: Removed LNI 192.168.206.111@tcp [17098.000363] LNet: Removed LNI 192.168.206.111@tcp10 [17098.007367] LNet: Removed LNI 192.168.206.111@tcp11 [17098.015214] LNet: Removed LNI 192.168.206.111@tcp12 [17098.022968] LNet: Removed LNI 192.168.206.111@tcp13 [17098.030485] LNet: Removed LNI 192.168.206.111@tcp14 [17098.036900] LNet: Removed LNI 192.168.206.111@tcp15 [17098.047672] LNet: Removed LNI 192.168.206.111@tcp16 [17098.055502] LNet: Removed LNI 192.168.206.111@tcp17 [17098.057593] LNet: Removed LNI 192.168.206.111@tcp18 [17098.078743] LNet: Removed LNI 192.168.206.111@tcp19 [17098.080941] LNet: Removed LNI 192.168.206.111@tcp20 [17098.091384] LNet: Removed LNI 192.168.206.111@tcp21 [17098.095475] LNet: Removed LNI 192.168.206.111@tcp22 [17098.099783] LNet: Removed LNI 192.168.206.111@tcp23 [17098.105230] LNet: Removed LNI 192.168.206.111@tcp24 [17098.111772] LNet: Removed LNI 192.168.206.111@tcp25 [17098.121579] LNet: Removed LNI 192.168.206.111@tcp26 [17098.132490] LNet: Removed LNI 192.168.206.111@tcp27 [17098.138558] LNet: Removed LNI 192.168.206.111@tcp28 [17098.143584] LNet: Removed LNI 192.168.206.111@tcp29 [17098.148131] LNet: Removed LNI 192.168.206.111@tcp30 [17098.163238] LNet: Removed LNI 192.168.206.111@tcp31 [17098.173264] LNet: Removed LNI 192.168.206.111@tcp32 [17098.180214] LNet: Removed LNI 192.168.206.111@tcp33 [17098.186872] LNet: Removed LNI 192.168.206.111@tcp34 [17098.191085] LNet: Removed LNI 192.168.206.111@tcp35 [17098.196234] LNet: Removed LNI 192.168.206.111@tcp36 [17098.198227] LNet: Removed LNI 192.168.206.111@tcp37 [17098.208445] LNet: Removed LNI 192.168.206.111@tcp38 [17098.215075] LNet: Removed LNI 192.168.206.111@tcp39 [17098.220541] LNet: Removed LNI 192.168.206.111@tcp40 [17098.225366] LNet: Removed LNI 192.168.206.111@tcp41 [17098.228814] LNet: Removed LNI 192.168.206.111@tcp42 [17098.233736] LNet: Removed LNI 192.168.206.111@tcp43 [17098.243561] LNet: Removed LNI 192.168.206.111@tcp44 [17098.249764] LNet: Removed LNI 192.168.206.111@tcp45 [17098.256609] LNet: Removed LNI 192.168.206.111@tcp46 [17098.279089] LNet: Removed LNI 192.168.206.111@tcp47 [17098.287226] LNet: Removed LNI 192.168.206.111@tcp48 [17098.294204] LNet: Removed LNI 192.168.206.111@tcp49 [17098.311917] LNet: Removed LNI 192.168.206.111@tcp50 [17099.609126] Key type .llcrypt unregistered [17099.612167] Key type ._llcrypt unregistered [17131.134856] Key type ._llcrypt registered [17131.137922] Key type .llcrypt registered [17131.419726] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [17155.699501] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [17158.896613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17159.067036] alg: No test for adler32 (adler32-zlib) [17160.484185] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [17161.125685] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [17162.984135] Key type lgssc registered [17164.616170] Lustre: Echo OBD driver; http://www.lustre.org/ [17210.788720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17210.850678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17212.522862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17212.598758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17212.735139] Lustre: lustre-MDT0000: new disk, initializing [17212.882041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17212.931568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17217.917851] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17232.132274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17232.252894] Lustre: 505986:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17232.346882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17232.359179] Lustre: Skipped 1 previous similar message [17232.493567] Lustre: lustre-MDT0001: new disk, initializing [17232.640017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17232.676065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17232.682590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17237.879171] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17251.303414] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17259.166457] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [17270.177478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17270.544973] Lustre: *** cfs_fail_loc=90f, val=0*** [17270.715443] Lustre: lustre-OST0000: new disk, initializing [17270.721738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17270.736486] Lustre: 508008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17270.871992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17279.957678] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17280.112504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17280.126919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17280.318532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17294.879907] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [17306.081777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17306.082379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17306.095055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17306.133641] Lustre: Skipped 1 previous similar message [17311.212643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17311.225756] Lustre: Skipped 1 previous similar message [17316.333990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17316.340580] Lustre: Skipped 1 previous similar message [17319.392975] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17319.680964] Lustre: server umount lustre-OST0000 complete [17325.033139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17325.045236] LustreError: Skipped 1 previous similar message [17325.049520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17325.060242] Lustre: Skipped 1 previous similar message [17325.070221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17326.569524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17330.949840] Lustre: server umount lustre-MDT0000 complete [17335.904133] LustreError: 505969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781600585 with bad export cookie 6181355857856110354 [17335.910746] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17335.918097] LustreError: 505969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17336.666826] Lustre: server umount lustre-MDT0001 complete [17352.013678] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [17355.556867] Key type lgssc unregistered [17355.911992] LNet: 509919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17355.921326] LNetError: 509919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [17355.939994] LNet: Removed LNI 192.168.206.111@tcp [17357.217847] Key type .llcrypt unregistered [17357.220449] Key type ._llcrypt unregistered [17385.733296] Key type ._llcrypt registered [17385.741573] Key type .llcrypt registered [17386.006453] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [17388.365190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17388.454330] alg: No test for adler32 (adler32-zlib) [17389.801073] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [17390.246756] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [17392.011873] Key type lgssc registered [17393.572235] Lustre: Echo OBD driver; http://www.lustre.org/ [17404.093466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17404.112090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17405.878577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17413.130869] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17416.197378] LustreError: 511192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17416.243525] LustreError: 511192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [17417.195312] LustreError: 511191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17422.308974] LustreError: 511192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17425.673767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17426.171841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17431.959663] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17440.437592] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17448.167263] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [17457.183705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17457.979683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17468.493629] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17481.001974] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [17494.077211] loop0: detected capacity change from 0 to 204800000 [17494.485818] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17494.521997] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17494.544485] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17503.258632] loop0: detected capacity change from 0 to 204800000 [17503.362971] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17503.372846] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17503.384714] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17510.328420] loop0: detected capacity change from 0 to 204800000 [17518.808285] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17518.966596] Lustre: fs15246-OST0000: new disk, initializing [17518.971465] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [17518.977461] Lustre: 514790:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17519.059988] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17528.539286] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17540.075870] loop1: detected capacity change from 0 to 204800000 [17546.231298] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17546.379849] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [17546.438293] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [17546.565049] Lustre: fs15246-MDT0000: new disk, initializing [17546.776824] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17546.810170] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17546.947368] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [17548.004765] Lustre: 515915:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000216e5559 x1868143671695616/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781600813 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [17552.242572] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17557.111490] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17557.133342] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17557.213144] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17580.011343] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17580.034113] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [17583.013377] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [17584.337738] Lustre: server umount fs15246-MDT0000 complete [17588.219216] LustreError: 515779:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17593.325669] LustreError: 512627:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17595.678982] Lustre: server umount fs15246-OST0000 complete [17609.699521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17609.714087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17609.740484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17610.736091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17610.753743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17613.803312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17613.811676] Lustre: Skipped 2 previous similar messages [17615.312907] Lustre: server umount lustre-OST0000 complete [17620.961582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17620.977198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17620.990983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17629.154167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17629.162286] Lustre: Skipped 3 previous similar messages [17634.272914] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17634.493060] Lustre: server umount lustre-MDT0000 complete [17638.370636] LustreError: 511191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17638.394377] LustreError: 511191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [17638.433260] LustreError: 516965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781600888 with bad export cookie 11132014996069181333 [17638.445903] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17638.447674] LustreError: 516965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17639.007130] Lustre: server umount lustre-MDT0001 complete [17652.408292] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [17656.112313] Key type lgssc unregistered [17656.547272] LNet: 519014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17656.559135] LNetError: 519014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [17656.574753] LNet: Removed LNI 192.168.206.111@tcp [17657.657218] Key type .llcrypt unregistered [17657.661229] Key type ._llcrypt unregistered [17680.941887] Key type ._llcrypt registered [17680.944177] Key type .llcrypt registered [17681.076101] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [17682.388924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17682.681409] alg: No test for adler32 (adler32-zlib) [17683.893114] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [17684.235251] LNet: Added LNI 192.168.206.111@tcp [8/256/0/180] [17685.995919] Key type lgssc registered [17687.237904] Lustre: Echo OBD driver; http://www.lustre.org/ [17731.124194] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [17743.175965] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [17786.795090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17786.818439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17788.183461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17788.229835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17788.367295] Lustre: lustre-MDT0000: new disk, initializing [17788.531929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17788.560329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17794.468505] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17807.556754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17807.791877] Lustre: 523481:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17807.934822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17807.945863] Lustre: Skipped 1 previous similar message [17808.156635] Lustre: lustre-MDT0001: new disk, initializing [17808.331777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17808.433341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17808.455501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17814.229872] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17825.363245] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17832.946738] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [17841.558103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17841.975930] Lustre: lustre-OST0000: new disk, initializing [17841.982863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17841.993330] Lustre: 525449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17842.140522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17850.845795] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_default_debug -1 all [17851.003243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17851.023959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17851.167470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17864.193075] Lustre: DEBUG MARKER: oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [17869.650869] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17869.955716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17876.158522] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17876.351622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17882.093256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17882.098455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17882.112166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17887.207403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17887.215373] Lustre: Skipped 2 previous similar messages [17892.328519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17892.336802] Lustre: Skipped 1 previous similar message [17893.344775] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17893.696987] Lustre: server umount lustre-OST0000 complete [17900.519738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17900.529520] LustreError: Skipped 1 previous similar message [17900.541314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17900.567916] Lustre: Skipped 1 previous similar message [17900.587200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17902.572872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17907.682669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17907.691556] Lustre: Skipped 3 previous similar messages [17913.312196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17913.515822] Lustre: server umount lustre-MDT0000 complete [17917.758807] LustreError: 523472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781601167 with bad export cookie 9359919329586310383 [17917.791397] LustreError: MGC192.168.206.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17918.227171] Lustre: server umount lustre-MDT0001 complete [17948.702132] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing set_hostid [17960.844407] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing load_modules_local [17997.922067] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 17592 sec ========= 05:14:06 (1781601246) [17999.795555] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:14:08 (1781601248) === [18003.595696] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:14:12 (1781601252) === [18047.411928] Lustre: DEBUG MARKER: oleg611-server.virtnet: executing unload_modules_local [18051.377649] Key type lgssc unregistered [18051.736523] LNet: 532102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18051.741678] LNetError: 532102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [18051.757478] LNet: Removed LNI 192.168.206.111@tcp [18052.785610] Key type .llcrypt unregistered [18052.787410] Key type ._llcrypt unregistered