[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1010988573 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2640MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003173] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.012010] LSM: Security Framework initializing [ 0.013054] Yama: becoming mindful. [ 0.014040] SELinux: Initializing. [ 0.015079] *** VALIDATE selinux *** [ 0.023524] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028056] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031152] *** VALIDATE tmpfs *** [ 0.032541] *** VALIDATE proc *** [ 0.033263] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035296] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038036] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042265] debug: unmapping init [mem 0xffffffffb7a59000-0xffffffffb7a60fff] [ 0.044194] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045748] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053341] rcu: Hierarchical SRCU implementation. [ 0.055621] smp: Bringing up secondary CPUs ... [ 0.056635] x86: Booting SMP configuration: [ 0.057033] .... node #0, CPUs: #1 #2 #3 [ 0.060422] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.142056] node 0 deferred pages initialised in 76ms [ 0.145151] devtmpfs: initialized [ 0.146222] x86/mm: Memory block size: 128MB [ 0.148891] gcov: version magic: 0x41383552 [ 0.150382] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155434] pinctrl core: initialized pinctrl subsystem [ 0.158174] [ 0.158808] ************************************************************* [ 0.161014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163011] ** ** [ 0.166012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168010] ** ** [ 0.170014] ** This means that this kernel is built to expose internal ** [ 0.173013] ** IOMMU data structures, which may compromise security on ** [ 0.175010] ** your system. ** [ 0.177012] ** ** [ 0.180057] ** If you see this message and you are not debugging the ** [ 0.182016] ** kernel, report this immediately to your vendor! ** [ 0.184052] ** ** [ 0.187012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189012] ************************************************************* [ 0.191778] NET: Registered protocol family 16 [ 0.194629] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203110] cpuidle: using governor menu [ 0.206246] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207524] PCI: Using configuration type 1 for base access [ 0.209156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227087] cryptd: max_cpu_qlen set to 1000 [ 0.229234] ACPI: Added _OSI(Module Device) [ 0.231014] ACPI: Added _OSI(Processor Device) [ 0.232012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239152] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246600] ACPI: Interpreter enabled [ 0.248071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250013] ACPI: Using IOAPIC for interrupt routing [ 0.252104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256412] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268588] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282500] acpiphp: Slot [2] registered [ 0.284126] acpiphp: Slot [5] registered [ 0.285097] acpiphp: Slot [6] registered [ 0.287104] acpiphp: Slot [7] registered [ 0.288142] acpiphp: Slot [8] registered [ 0.290101] acpiphp: Slot [9] registered [ 0.291129] acpiphp: Slot [10] registered [ 0.293108] acpiphp: Slot [3] registered [ 0.294105] acpiphp: Slot [4] registered [ 0.296112] acpiphp: Slot [11] registered [ 0.297135] acpiphp: Slot [12] registered [ 0.299160] acpiphp: Slot [13] registered [ 0.300095] acpiphp: Slot [14] registered [ 0.302128] acpiphp: Slot [15] registered [ 0.303082] acpiphp: Slot [16] registered [ 0.305152] acpiphp: Slot [17] registered [ 0.306123] acpiphp: Slot [18] registered [ 0.307090] acpiphp: Slot [19] registered [ 0.309098] acpiphp: Slot [20] registered [ 0.310095] acpiphp: Slot [21] registered [ 0.312115] acpiphp: Slot [22] registered [ 0.313087] acpiphp: Slot [23] registered [ 0.315083] acpiphp: Slot [24] registered [ 0.316145] acpiphp: Slot [25] registered [ 0.318105] acpiphp: Slot [26] registered [ 0.319161] acpiphp: Slot [27] registered [ 0.321106] acpiphp: Slot [28] registered [ 0.322103] acpiphp: Slot [29] registered [ 0.324107] acpiphp: Slot [30] registered [ 0.325131] acpiphp: Slot [31] registered [ 0.327062] PCI host bridge to bus 0000:00 [ 0.328021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.344187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348106] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351522] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.361805] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366669] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377573] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380875] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.383046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.386913] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.391016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.415031] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.435019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.461019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.480020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.505863] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.544021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.575024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.677022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.692184] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.729024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.762021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.829021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.868000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.880186] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.899028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.929020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.951754] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.971020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.989021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.043019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.066551] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.102025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.140022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.244024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.285678] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.288384] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.290417] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.293413] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.295287] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.301132] iommu: Default domain type: Passthrough [ 1.303641] SCSI subsystem initialized [ 1.305185] ACPI: bus type USB registered [ 1.307134] usbcore: registered new interface driver usbfs [ 1.309112] usbcore: registered new interface driver hub [ 1.311102] usbcore: registered new device driver usb [ 1.313150] pps_core: LinuxPPS API ver. 1 registered [ 1.315013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.318072] PTP clock support registered [ 1.320073] EDAC MC: Ver: 3.0.0 [ 1.322126] PCI: Using ACPI for IRQ routing [ 1.323982] NetLabel: Initializing [ 1.326014] NetLabel: domain hash size = 128 [ 1.327011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.329094] NetLabel: unlabeled traffic allowed by default [ 1.332046] vgaarb: loaded [ 1.333304] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.335015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.342000] clocksource: Switched to clocksource kvm-clock [ 1.456166] VFS: Disk quotas dquot_6.6.0 [ 1.457595] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.459906] *** VALIDATE ramfs *** [ 1.461074] *** VALIDATE hugetlbfs *** [ 1.462450] pnp: PnP ACPI init [ 1.464617] pnp: PnP ACPI: found 6 devices [ 1.479595] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.482646] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.484666] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.486605] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.489062] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.491636] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.494562] NET: Registered protocol family 2 [ 1.496843] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.501788] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.505256] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.510427] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.513866] TCP: Hash tables configured (established 65536 bind 65536) [ 1.517294] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.520481] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.523413] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.526537] NET: Registered protocol family 1 [ 1.529469] RPC: Registered named UNIX socket transport module. [ 1.531457] RPC: Registered udp transport module. [ 1.532889] RPC: Registered tcp transport module. [ 1.534828] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.536945] NET: Registered protocol family 44 [ 1.538743] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.540959] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.543378] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.545808] PCI: CLS 0 bytes, default 64 [ 1.547993] Unpacking initramfs... [ 3.201266] debug: unmapping init [mem 0xffff8d613cc54000-0xffff8d613ffbffff] [ 3.205315] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.207432] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 3.210267] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 3.714990] Initialise system trusted keyrings [ 3.716395] Key type blacklist registered [ 3.717968] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.726461] zbud: loaded [ 3.729068] *** VALIDATE nfs *** [ 3.729964] *** VALIDATE nfs4 *** [ 3.731650] pstore: using deflate compression [ 3.734885] Platform Keyring initialized [ 3.837317] NET: Registered protocol family 38 [ 3.839165] Key type asymmetric registered [ 3.841053] Asymmetric key parser 'x509' registered [ 3.845486] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.849562] io scheduler mq-deadline registered [ 3.850925] io scheduler kyber registered [ 3.853704] io scheduler bfq registered [ 3.856165] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.858878] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.862246] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.865290] ACPI: Power Button [PWRF] [ 3.870899] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.877824] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.943529] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.953049] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.976896] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.006614] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.034755] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.039596] Non-volatile memory driver v1.3 [ 4.041170] Linux agpgart interface v0.103 [ 4.073609] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 4.076697] vda: detected capacity change from 0 to 76775424 [ 4.097396] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.100328] vdb: detected capacity change from 0 to 1073741824 [ 4.121254] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.125626] vdc: detected capacity change from 0 to 2621440000 [ 4.158237] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.161301] vdd: detected capacity change from 0 to 2621440000 [ 4.178164] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.181019] vde: detected capacity change from 0 to 4294967296 [ 4.199279] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.202799] vdf: detected capacity change from 0 to 4294967296 [ 4.214846] libphy: Fixed MDIO Bus: probed [ 4.231781] usbcore: registered new interface driver usbserial_generic [ 4.235380] usbserial: USB Serial support registered for generic [ 4.237694] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.242583] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.244508] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.247253] mousedev: PS/2 mouse device common for all mice [ 4.250172] rtc_cmos 00:05: RTC can wake from S4 [ 4.255994] rtc_cmos 00:05: registered as rtc0 [ 4.257580] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.260279] intel_pstate: CPU model not supported [ 4.262287] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.266841] hid: raw HID events driver (C) Jiri Kosina [ 4.268807] usbcore: registered new interface driver usbhid [ 4.270819] usbhid: USB HID core driver [ 4.272576] drop_monitor: Initializing network drop monitor service [ 4.275278] Initializing XFRM netlink socket [ 4.278615] NET: Registered protocol family 10 [ 4.282369] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.283618] Segment Routing with IPv6 [ 4.289901] NET: Registered protocol family 17 [ 4.292520] mpls_gso: MPLS GSO support [ 4.301154] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.307354] RAS: Correctable Errors collector initialized. [ 4.309704] AVX version of gcm_enc/dec engaged. [ 4.311346] AES CTR mode by8 optimization enabled [ 4.401520] sched_clock: Marking stable (4401440526, 0)->(5336595225, -935154699) [ 4.405849] registered taskstats version 1 [ 4.407916] Loading compiled-in X.509 certificates [ 4.410247] zswap: loaded using pool lzo/zbud [ 4.437492] Key type big_key registered [ 4.449675] Key type encrypted registered [ 4.453182] ima: No TPM chip found, activating TPM-bypass! [ 4.454900] ima: Allocated hash algorithm: sha1 [ 4.456334] ima: No architecture policies found [ 4.457622] evm: Initialising EVM extended attributes: [ 4.459150] evm: security.selinux [ 4.459924] evm: security.ima [ 4.460803] evm: security.capability [ 4.461795] evm: HMAC attrs: 0x1 [ 4.464181] rtc_cmos 00:05: setting system clock to 2026-09-14 06:15:58 UTC (1789366558) [ 4.469583] debug: unmapping init [mem 0xffffffffb8a03000-0xffffffffb8bfffff] [ 4.472653] debug: unmapping init [mem 0xffffffffb7782000-0xffffffffb7a58fff] [ 4.482148] Write protecting the kernel read-only data: 28672k [ 4.485700] debug: unmapping init [mem 0xffffffffb5e03000-0xffffffffb5ffffff] [ 4.488695] debug: unmapping init [mem 0xffffffffb6714000-0xffffffffb67fffff] [ 4.524329] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.533820] systemd[1]: Detected virtualization kvm. [ 4.536213] systemd[1]: Detected architecture x86-64. [ 4.538386] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.565365] systemd[1]: No hostname configured. [ 4.567480] systemd[1]: Set hostname to . [ 4.569569] random: systemd: uninitialized urandom read (16 bytes read) [ 4.572266] systemd[1]: Initializing machine ID from random generator. [ 4.715630] random: systemd: uninitialized urandom read (16 bytes read) [ 4.718485] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.724658] random: systemd: uninitialized urandom read (16 bytes read) [ 4.727798] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.735778] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.504392] device-mapper: uevent: version 1.0.3 [ 5.509861] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 6.727764] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.762896] random: fast init done Starting dracut initqueue hook... [ 6.914556] scsi host0: ata_piix [ 7.709844] scsi host1: ata_piix [ 7.715921] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.721814] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.980273] random: crng init done [ 13.981692] random: 7 urandom warning(s) missed due to ratelimiting [ 17.133492] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.175623] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.460429] printk: systemd: 25 output lines suppressed due to ratelimiting [ 21.118201] SELinux: Disabled at runtime. [ 21.177809] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.185184] systemd[1]: Detected virtualization kvm. [ 21.186783] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.211487] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.218310] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.228890] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.234145] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.238613] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.249059] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.256883] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward [ 22.348576] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.526794] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.081319] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.510250] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.381046] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.421176] EDAC sbridge: Ver: 1.1.2 [ 28.440690] Key type dns_resolver registered [ 28.820793] NFS: Registering the id_resolver key type [ 28.823361] Key type id_resolver registered [ 28.825296] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg104-server login: [ 108.536922] libcfs: loading out-of-tree module taints kernel. [ 108.562540] Key type ._llcrypt registered [ 108.564250] Key type .llcrypt registered [ 108.640890] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 123.361061] hrtimer: interrupt took 7496540 ns [ 124.399511] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 126.036212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 126.061210] alg: No test for adler32 (adler32-zlib) [ 127.407724] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 128.204277] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 129.975265] Key type lgssc registered [ 131.182518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.279699] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.669642] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 198.059153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.084895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.426549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.462889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.616563] Lustre: lustre-MDT0000: new disk, initializing [ 199.785635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.829251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 203.864200] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.138369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.246841] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.272127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.275694] Lustre: Skipped 1 previous similar message [ 217.337359] Lustre: lustre-MDT0001: new disk, initializing [ 217.371777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.391846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.419911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.636937] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.449632] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 237.064683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.350826] Lustre: lustre-OST0000: new disk, initializing [ 237.354526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 237.360762] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.423828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.012509] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.328827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 247.337732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.430405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 258.049563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.177714] Lustre: lustre-OST0001: new disk, initializing [ 258.181790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 258.188917] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.221424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.613354] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.833216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 265.849303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 265.919764] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 276.864590] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.476367] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.189827] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing check_logdir /tmp/testlogs/ [ 298.268415] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing yml_node [ 303.479521] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 305.794652] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 308.978873] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 310.683685] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 02:21:02 EDT 2026 [ 327.055556] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 328.928228] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.579382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 345.585742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.595079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.128542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.128784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.133028] Lustre: Skipped 1 previous similar message [ 348.150385] Lustre: Skipped 1 previous similar message [ 353.251503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.263374] Lustre: Skipped 5 previous similar messages [ 358.368730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.380767] Lustre: Skipped 3 previous similar messages [ 359.391160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 359.875614] Lustre: server umount lustre-MDT0000 complete [ 363.210546] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789366917 with bad export cookie 16955274604391249496 [ 363.210794] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.231914] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 363.489173] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.491602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.513078] Lustre: Skipped 2 previous similar messages [ 368.609144] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.624409] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 369.625208] Lustre: server umount lustre-MDT0001 complete [ 373.620721] Lustre: server umount lustre-OST0000 complete [ 377.779392] Lustre: server umount lustre-OST0001 complete [ 384.412529] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 394.294306] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 429.206456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.397487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 429.410753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 429.471295] Lustre: lustre-MDT0000: new disk, initializing [ 429.531679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.548621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 434.320990] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 444.420450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.537704] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 444.587114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 444.589580] Lustre: Skipped 1 previous similar message [ 444.632363] Lustre: lustre-MDT0001: new disk, initializing [ 444.693536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 444.730930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 444.744691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 449.120518] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 458.097195] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 463.795356] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 469.240834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 469.429349] Lustre: lustre-OST0000: new disk, initializing [ 469.432793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 469.437839] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 469.520896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.670790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 470.692369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 470.758607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 474.744360] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 482.915979] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 487.067796] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 487.252535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 491.583166] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 491.760929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 496.613914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.617194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.627018] Lustre: Skipped 1 previous similar message [ 496.633839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.633849] Lustre: Skipped 3 previous similar messages [ 496.666748] LustreError: Skipped 1 previous similar message [ 499.921274] Lustre: server umount lustre-OST0000 complete [ 506.336357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.344256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.860256] Lustre: server umount lustre-MDT0000 complete [ 511.970603] LustreError: 15081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.985349] LustreError: 15081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 513.211876] LustreError: 18581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367067 with bad export cookie 16955274604391251036 [ 513.214372] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.220129] LustreError: 18581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 513.607272] Lustre: server umount lustre-MDT0001 complete [ 517.842921] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:24:29 (1789367069) [ 519.784839] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 521.152842] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:24:33 (1789367073) [ 527.167110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.592551] LustreError: 19451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.691388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.330710] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 532.963968] LustreError: 19452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.059614] LustreError: 19451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.262448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.363726] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 551.629339] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 556.949748] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 564.476128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.901737] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 577.418259] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 587.233456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 587.238456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.246636] Lustre: Skipped 1 previous similar message [ 587.251043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 587.254590] Lustre: Skipped 3 previous similar messages [ 590.516110] Lustre: server umount lustre-OST0000 complete [ 597.993314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 598.007633] LustreError: Skipped 1 previous similar message [ 608.223184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 608.363257] Lustre: server umount lustre-MDT0000 complete [ 608.739745] LustreError: 20509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.749585] LustreError: 20509:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 611.179466] LustreError: 19433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367165 with bad export cookie 16955274604391251918 [ 611.182092] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.190080] LustreError: 19433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.488724] Lustre: server umount lustre-MDT0001 complete [ 620.603794] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 623.199600] Key type lgssc unregistered [ 623.548051] LNet: 22981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.562583] LNetError: 22981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.583231] LNet: Removed LNI 192.168.201.104@tcp [ 624.479158] Key type .llcrypt unregistered [ 624.482197] Key type ._llcrypt unregistered [ 643.490830] Key type ._llcrypt registered [ 643.495420] Key type .llcrypt registered [ 643.606315] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 644.925604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 644.949848] alg: No test for adler32 (adler32-zlib) [ 646.013663] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 646.268227] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 647.904194] Key type lgssc registered [ 648.902327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 656.302613] loop: module loaded [ 656.396239] loop0: detected capacity change from 0 to 204800000 [ 656.566691] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.575978] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.772751] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.089918] loop0: detected capacity change from 0 to 204800000 [ 662.282290] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.295959] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.305379] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 666.800969] loop0: detected capacity change from 0 to 204800000 [ 671.587937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 671.601307] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.840560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 672.882486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 672.949689] Lustre: lustre-MDT0000: new disk, initializing [ 673.010422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 673.024431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 677.112330] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 684.491065] loop1: detected capacity change from 0 to 204800000 [ 689.495329] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 689.744255] Lustre: lustre-OST07c6: new disk, initializing [ 689.746392] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 689.748363] Lustre: Skipped 1 previous similar message [ 689.751243] Lustre: 26146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 689.807187] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 695.065762] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 698.931428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 698.936256] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 698.974090] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 704.886852] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 704.891508] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 704.904685] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 704.913230] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 704.917483] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 704.922713] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 717.571914] Lustre: 26153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 717.577745] Lustre: 26153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 717.583244] Lustre: 26153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.587804] Lustre: 26153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 717.593675] Lustre: 26153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 717.596764] Lustre: 26153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.019843] Lustre: 26153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 722.024734] Lustre: 26153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.032847] Lustre: 26153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 722.037595] Lustre: 26153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.046622] Lustre: 26153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.051390] Lustre: 26153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.056635] Lustre: 26153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 722.062736] Lustre: 26153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.070796] Lustre: 26153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 722.076824] Lustre: 26153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.083839] Lustre: 26153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.092156] Lustre: 26153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.254430] Lustre: 26153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 725.264188] Lustre: 26153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.286207] Lustre: 26153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.296981] Lustre: 26153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.305812] Lustre: 26153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.317020] Lustre: 26153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.328310] Lustre: 26153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 725.339224] Lustre: 26153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.350294] Lustre: 26153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.355979] Lustre: 26153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.367187] Lustre: 26153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.375358] Lustre: 26153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.521743] Lustre: 25131:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 729.528199] Lustre: 25131:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 729.533414] Lustre: 25131:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 729.541740] Lustre: 25131:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.550438] Lustre: 25131:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 729.562301] Lustre: 25131:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.569495] Lustre: 25131:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 729.584238] Lustre: 25131:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.597300] Lustre: 25131:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 729.603511] Lustre: 25131:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.610666] Lustre: 25131:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 729.617932] Lustre: 25131:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.091659] Lustre: 26838:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 740.097427] Lustre: 26838:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 740.101308] Lustre: 26838:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 740.105220] Lustre: 26838:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 740.109543] Lustre: 26838:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 740.114594] Lustre: 26838:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 740.119154] Lustre: 26838:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 740.128049] Lustre: 26838:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 740.132700] Lustre: 26838:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 740.137469] Lustre: 26838:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 740.142076] Lustre: 26838:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.146147] Lustre: 26838:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.013323] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.023307] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 791.025301] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 793.722234] Lustre: server umount lustre-OST07c6 complete [ 801.364511] Lustre: server umount lustre-MDT0000 complete [ 814.197482] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 816.638311] Key type lgssc unregistered [ 816.851918] LNet: 28698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.865882] LNetError: 28698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.918342] LNet: Removed LNI 192.168.201.104@tcp [ 817.740619] Key type .llcrypt unregistered [ 817.742837] Key type ._llcrypt unregistered [ 832.624347] Key type ._llcrypt registered [ 832.626746] Key type .llcrypt registered [ 832.710181] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 833.962829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.979143] alg: No test for adler32 (adler32-zlib) [ 835.083962] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 835.279342] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 836.935326] Key type lgssc registered [ 837.778218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 843.437196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 843.446127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.783655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.114108] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 853.985784] LustreError: 29945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 853.996781] LustreError: 29945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 855.077951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.434141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 858.799560] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 864.651992] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 869.040425] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 874.203083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.486508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.005995] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 884.495507] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 889.844215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 901.097295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 901.108489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.117473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 905.193456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.198036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 905.198211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 906.552701] Lustre: server umount lustre-OST0000 complete [ 910.309511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.316501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 910.329955] Lustre: Skipped 1 previous similar message [ 910.343208] Lustre: Skipped 2 previous similar messages [ 915.431038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 915.659641] Lustre: server umount lustre-MDT0000 complete [ 919.084988] LustreError: 29925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367473 with bad export cookie 11428721966676576166 [ 919.097132] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.329159] Lustre: server umount lustre-MDT0001 complete [ 927.989336] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 930.318651] Key type lgssc unregistered [ 930.615875] LNet: 33678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.620104] LNetError: 33678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.686774] LNet: Removed LNI 192.168.201.104@tcp [ 932.472206] Key type .llcrypt unregistered [ 932.473677] Key type ._llcrypt unregistered [ 948.071778] Key type ._llcrypt registered [ 948.073723] Key type .llcrypt registered [ 948.136747] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 948.928966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.948161] alg: No test for adler32 (adler32-zlib) [ 949.902514] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 950.054334] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 951.719527] Key type lgssc registered [ 952.750249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 959.217490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 959.235113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.561504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.018837] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 970.722022] LustreError: 34924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.727983] LustreError: 34924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 970.934191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.177617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 974.976322] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 982.323850] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 988.144534] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 994.083856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.480410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.613397] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1005.593680] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1005.749975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1016.291851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.302204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1016.304706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.317025] Lustre: Skipped 1 previous similar message [ 1021.414126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.425952] Lustre: Skipped 1 previous similar message [ 1021.813108] Lustre: server umount lustre-OST0000 complete [ 1026.529263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.530587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.556369] Lustre: Skipped 2 previous similar messages [ 1026.570332] Lustre: Skipped 1 previous similar message [ 1031.649673] LustreError: 34925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.778339] Lustre: server umount lustre-MDT0000 complete [ 1034.601952] LustreError: 34905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367588 with bad export cookie 4308203438886291151 [ 1034.611383] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.615649] LustreError: 34905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1034.897256] Lustre: server umount lustre-MDT0001 complete [ 1043.122765] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1045.304748] Key type lgssc unregistered [ 1045.505589] LNet: 38457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.511238] LNetError: 38457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.525782] LNet: Removed LNI 192.168.201.104@tcp [ 1046.136393] Key type .llcrypt unregistered [ 1046.138225] Key type ._llcrypt unregistered [ 1062.155311] Key type ._llcrypt registered [ 1062.163358] Key type .llcrypt registered [ 1062.238370] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1062.724470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1062.790055] alg: No test for adler32 (adler32-zlib) [ 1063.741667] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1063.906220] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1065.575412] Key type lgssc registered [ 1066.530985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1071.736899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.750268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.994783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.546586] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1082.420065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.658853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1085.735144] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1091.370439] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1095.604678] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1100.453018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1100.688186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.129867] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1110.864665] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1119.151118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1127.596434] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.4@tcp (stopping) [ 1128.313324] Lustre: server umount lustre-MDT0000 complete [ 1128.929541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1128.940892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.954037] LustreError: 39848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.974229] LustreError: 39848:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1129.445226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.457523] Lustre: Skipped 1 previous similar message [ 1129.462919] LustreError: 39845:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.472336] LustreError: 39845:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1131.052698] LustreError: 42443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367685 with bad export cookie 12993931708719312036 [ 1131.054033] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1131.059918] LustreError: 42443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1132.698150] LustreError: 42335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1132.711478] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.4@tcp (stopping) [ 1133.343356] Lustre: server umount lustre-MDT0001 complete [ 1145.322506] LustreError: 42443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.4@tcp arrived at 1789367699 with bad export cookie 12993931708719312701 [ 1145.333996] LustreError: 42443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1163.604868] Lustre: server umount lustre-OST0000 complete [ 1175.248181] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1177.402885] Key type lgssc unregistered [ 1177.667315] LNet: 43727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.674338] LNetError: 43727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.704567] LNet: Removed LNI 192.168.201.104@tcp [ 1178.466164] Key type .llcrypt unregistered [ 1178.468590] Key type ._llcrypt unregistered [ 1197.641485] Key type ._llcrypt registered [ 1197.644606] Key type .llcrypt registered [ 1197.696921] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1198.554140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.623916] alg: No test for adler32 (adler32-zlib) [ 1199.567094] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1199.748262] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1201.391317] Key type lgssc registered [ 1202.246719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1208.229754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1208.242830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.567938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1212.711957] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1218.645041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.853158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1221.950403] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1227.071725] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1230.969325] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1235.218049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1235.434202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.905612] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1243.456776] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1254.887444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1258.981432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1258.986932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.996044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1260.005897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1260.006222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.025714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1265.121202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1265.126560] Lustre: Skipped 3 previous similar messages [ 1270.244780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1270.247772] Lustre: Skipped 1 previous similar message [ 1273.311272] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1273.413293] Lustre: server umount lustre-OST0000 complete [ 1290.720902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.725103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1290.728665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.728673] Lustre: Skipped 1 previous similar message [ 1290.732242] Lustre: Skipped 1 previous similar message [ 1294.432422] Lustre: server umount lustre-MDT0000 complete [ 1295.844472] LustreError: 45845:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.856156] LustreError: 45845:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1296.708452] LustreError: 45100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367850 with bad export cookie 16238224946360870595 [ 1296.714540] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.715836] LustreError: 45100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1311.199125] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1311.378308] Lustre: server umount lustre-MDT0001 complete [ 1318.467666] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1320.269410] Key type lgssc unregistered [ 1320.465420] LNet: 48844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1320.469492] LNetError: 48844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1320.478584] LNet: Removed LNI 192.168.201.104@tcp [ 1321.077985] Key type .llcrypt unregistered [ 1321.079646] Key type ._llcrypt unregistered [ 1336.088579] Key type ._llcrypt registered [ 1336.090738] Key type .llcrypt registered [ 1336.171247] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1336.747185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1336.839852] alg: No test for adler32 (adler32-zlib) [ 1337.761384] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1337.906900] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1339.536933] Key type lgssc registered [ 1340.235905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1345.019784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1345.031397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.321923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.949957] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1354.161432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.398877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.193796] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1361.394217] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1364.475396] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1368.398536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1368.590989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.698547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1371.963971] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1375.561915] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1383.910868] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1385.648940] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1396.942366] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-14 2h39m09s [ 1397.843368] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1400.291554] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.302755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1400.314373] Lustre: Skipped 2 previous similar messages [ 1400.317527] Lustre: Skipped 1 previous similar message [ 1401.377660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.387079] Lustre: Skipped 3 previous similar messages [ 1403.557814] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.4@tcp (stopping) [ 1405.432678] Lustre: server umount lustre-MDT0000 complete [ 1407.330652] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1407.458771] LustreError: 50231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.467244] LustreError: 50231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1409.633299] LustreError: 50233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.098446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.194281] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1411.345827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.382023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1412.391628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1413.839624] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1415.575287] Lustre: DEBUG MARKER: Wait for df (36627) ... [ 1416.330986] Lustre: DEBUG MARKER: done [ 1420.771310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.775089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.780962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.783668] Lustre: Skipped 1 previous similar message [ 1421.792621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1421.799484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.115738] Lustre: server umount lustre-OST0000 complete [ 1427.938720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1427.939604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.946262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.960584] Lustre: Skipped 1 previous similar message [ 1427.965654] Lustre: Skipped 4 previous similar messages [ 1432.440870] Lustre: server umount lustre-MDT0000 complete [ 1433.060893] LustreError: 50237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.077084] LustreError: 50237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1434.813247] LustreError: 50217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367988 with bad export cookie 1808762626045997306 [ 1434.816551] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.821086] LustreError: 50217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1435.039728] Lustre: server umount lustre-MDT0001 complete [ 1441.741189] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1443.311461] Key type lgssc unregistered [ 1443.506572] LNet: 55147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.519430] LNetError: 55147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.587447] LNet: Removed LNI 192.168.201.104@tcp [ 1445.102357] Key type .llcrypt unregistered [ 1445.104893] Key type ._llcrypt unregistered [ 1468.876089] Key type ._llcrypt registered [ 1468.881199] Key type .llcrypt registered [ 1468.964537] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1469.610050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.643215] alg: No test for adler32 (adler32-zlib) [ 1470.598154] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1470.754410] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1472.367206] Key type lgssc registered [ 1473.110565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.951772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1477.967393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.126321] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1479.139929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1479.228749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.087068] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1487.677224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.737907] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1487.764072] Lustre: 57330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1487.932698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.381212] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1495.366983] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1499.286506] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1503.842500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1503.945637] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1504.086825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1508.499475] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1512.049886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1513.486626] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1517.975517] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-14 2h41m10s [ 1518.946761] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1523.260540] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1525.018873] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1525.071772] LustreError: 57338:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1545.404770] Lustre: lustre-MDT0000: Client 09ee82db-3c16-4f82-a460-93a32cb36254 (at 192.168.201.4@tcp) reconnecting [ 1565.127161] LustreError: 57338:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1566.584657] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1568.287421] Lustre: DEBUG MARKER: done [ 1573.353577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1573.360443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1573.371317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1573.857328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1573.862086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1573.872586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1578.085345] Lustre: server umount lustre-OST0000 complete [ 1584.096547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1584.100063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.106482] Lustre: Skipped 1 previous similar message [ 1584.114210] Lustre: Skipped 1 previous similar message [ 1586.622193] Lustre: server umount lustre-MDT0000 complete [ 1588.951599] LustreError: 57324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368142 with bad export cookie 7137872790585422543 [ 1588.952974] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1588.966398] LustreError: 57324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1589.130254] Lustre: server umount lustre-MDT0001 complete [ 1596.269978] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1598.025846] Key type lgssc unregistered [ 1598.187741] LNet: 61656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.194848] LNetError: 61656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.212925] LNet: Removed LNI 192.168.201.104@tcp [ 1598.848104] Key type .llcrypt unregistered [ 1598.851459] Key type ._llcrypt unregistered [ 1625.163364] Key type ._llcrypt registered [ 1625.165121] Key type .llcrypt registered [ 1625.230095] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1625.656286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.669955] alg: No test for adler32 (adler32-zlib) [ 1626.602384] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1626.797190] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1628.415191] Key type lgssc registered [ 1629.067506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1630.911644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1630.922847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.072988] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1632.084962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1632.164417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.762737] Lustre: Failing over lustre-MDT0000 [ 1632.932712] Lustre: server umount lustre-MDT0000 complete [ 1637.012740] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:43:09 (1789368189) [ 1641.300686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.525865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.066612] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1649.162855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1649.218517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1649.239344] Lustre: 64425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1649.407374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1652.052877] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1655.946665] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1659.020261] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1663.100547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1663.205868] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1663.298509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.902926] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1670.201676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1671.335271] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1676.233889] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1680.355575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.355959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1680.357116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1680.363714] Lustre: Skipped 1 previous similar message [ 1685.482980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1685.488404] Lustre: Skipped 4 previous similar messages [ 1685.751943] Lustre: server umount lustre-MDT0000 complete [ 1687.886718] LustreError: 64417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368241 with bad export cookie 9670344093958693090 [ 1687.889286] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.893727] LustreError: 64417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1688.083398] Lustre: server umount lustre-MDT0001 complete [ 1690.616424] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1693.500906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1698.831480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.975195] Lustre: 63351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368244/real 1789368244] req@0000000096d88855 x1876288528746112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368260 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1707.001402] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.009389] Lustre: Skipped 1 previous similar message [ 1708.767151] Lustre: 63352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368246/real 1789368246] req@00000000ee7e0e2e x1876288528746240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368262 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1711.072179] Lustre: 63350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368249/real 1789368249] req@00000000d3446c27 x1876288528746368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1712.162907] LustreError: 63348:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008d2a6319 x1876288528747136/t0(0) o250->MGC192.168.201.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1712.420853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.479144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1713.439741] Lustre: 63350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368251/real 1789368251] req@000000006e030f6f x1876288528746624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368267 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1714.966637] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1717.474386] LustreError: 68194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1717.486060] LustreError: 68194:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1718.496060] Lustre: 63351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368256/real 1789368256] req@000000006c563d7c x1876288528746752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368272 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1720.000517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.426944] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1725.426800] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1726.534651] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1729.486812] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1735.650483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1735.654041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.661831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1735.677029] Lustre: Skipped 1 previous similar message [ 1738.999277] Lustre: server umount lustre-MDT0000 complete [ 1741.025146] LustreError: 66280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368295 with bad export cookie 9670344093958695365 [ 1741.027300] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1741.033951] LustreError: 66280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1741.203395] Lustre: server umount lustre-MDT0001 complete [ 1744.998668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1750.355269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.008562] LustreError: 63348:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ed046d7e x1876288528768384/t0(0) o250->MGC192.168.201.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1751.183383] LustreError: 66274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1751.230412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1751.234515] Lustre: Skipped 1 previous similar message [ 1751.261571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1753.679631] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1756.646068] LustreError: 66273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.656623] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.344547] Lustre: 63350:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368295/real 1789368295] req@00000000eb756b52 x1876288528768128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1758.806959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.418933] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1764.328610] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1765.680925] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1766.381899] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1768.805020] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1774.561709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1774.565197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.575850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1774.579411] Lustre: Skipped 4 previous similar messages [ 1778.856549] Lustre: server umount lustre-MDT0000 complete [ 1781.231684] LustreError: 64418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368335 with bad export cookie 9670344093958696737 [ 1781.235331] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1781.247540] LustreError: 64418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1781.430703] Lustre: server umount lustre-MDT0001 complete [ 1785.041517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1786.922573] Lustre: DEBUG MARKER: files compared the same [ 1797.193401] Lustre: server umount lustre-OST0000 complete [ 1805.307593] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1806.958573] Key type lgssc unregistered [ 1807.170094] LNet: 74141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.177202] LNetError: 74141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.188618] LNet: Removed LNI 192.168.201.104@tcp [ 1807.662936] Key type .llcrypt unregistered [ 1807.665097] Key type ._llcrypt unregistered [ 1821.277690] Key type ._llcrypt registered [ 1821.280794] Key type .llcrypt registered [ 1821.347734] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1822.001718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1822.037554] alg: No test for adler32 (adler32-zlib) [ 1823.002356] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1823.191135] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1824.824046] Key type lgssc registered [ 1825.589086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1830.389612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1830.400343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.700418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.270185] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug +malloc all [ 1839.597603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.757592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1841.831354] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug +malloc all [ 1845.733137] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1849.068846] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1852.705271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1852.903209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.961477] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug +malloc all [ 1855.973303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1859.937540] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1871.328110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1871.332792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.344519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1871.747259] Lustre: server umount lustre-OST0000 complete [ 1874.400839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.408088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.414534] Lustre: Skipped 1 previous similar message [ 1874.421980] Lustre: Skipped 1 previous similar message [ 1879.521678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.531987] Lustre: Skipped 1 previous similar message [ 1880.358450] Lustre: server umount lustre-MDT0000 complete [ 1882.457917] LustreError: 75514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368436 with bad export cookie 10993713418191833587 [ 1882.464287] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1882.472657] LustreError: 75514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1892.933283] Lustre: server umount lustre-MDT0001 complete [ 1898.371321] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1899.875489] Key type lgssc unregistered [ 1900.054546] LNet: 79058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.058802] LNetError: 79058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.068482] LNet: Removed LNI 192.168.201.104@tcp [ 1900.515692] Key type .llcrypt unregistered [ 1900.517700] Key type ._llcrypt unregistered [ 1913.371176] Key type ._llcrypt registered [ 1913.373019] Key type .llcrypt registered [ 1913.440830] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1914.096504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1914.122872] alg: No test for adler32 (adler32-zlib) [ 1915.022684] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 1915.174746] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1916.783512] Key type lgssc registered [ 1917.316536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1921.209076] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1921.218617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1936.735350] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1941.855624] LustreError: 80423:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.104@tcp: failed processing log, type 4: rc = -110 [ 1968.480276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.453574] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1974.811938] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1977.811797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.858989] Lustre: *** cfs_fail_loc=706, val=0*** [ 1978.001668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.028674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1979.551805] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1983.039457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.137897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1983.141113] Lustre: Skipped 1 previous similar message [ 1983.212545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.072375] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1988.051765] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1990.346422] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1993.699112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1993.700772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.702527] LustreError: Skipped 1 previous similar message [ 1993.708331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1993.712198] Lustre: Skipped 1 previous similar message [ 1997.421268] Lustre: server umount lustre-OST0000 complete [ 2003.936779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.937392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2003.942071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.943059] Lustre: Skipped 2 previous similar messages [ 2005.280428] Lustre: server umount lustre-MDT0000 complete [ 2007.009575] LustreError: 83081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368561 with bad export cookie 731520475031059676 [ 2007.012191] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.015708] LustreError: 83081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2007.161451] Lustre: server umount lustre-MDT0001 complete [ 2011.656055] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2013.016789] Key type lgssc unregistered [ 2013.166481] LNet: 84032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.170407] LNetError: 84032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.179311] LNet: Removed LNI 192.168.201.104@tcp [ 2013.605785] Key type .llcrypt unregistered [ 2013.607549] Key type ._llcrypt unregistered [ 2025.113164] Key type ._llcrypt registered [ 2025.114319] Key type .llcrypt registered [ 2025.159125] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2025.513104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2025.555371] alg: No test for adler32 (adler32-zlib) [ 2026.420323] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2026.517257] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2028.111215] Key type lgssc registered [ 2028.532917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2031.508679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2031.516482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.018554] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2036.918260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.142738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2038.681692] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2042.261284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.337066] LustreError: 86105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.344405] LustreError: 86105:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2042.404684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2044.631837] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2047.457030] LustreError: 86108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.463598] LustreError: 86108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2047.973026] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.978904] Lustre: MGS: Client 14634ccd-9a3d-4224-b62e-2f79f025233a (at 0@lo) reconnecting [ 2047.990476] LustreError: 86119:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2048.088305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.144297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2050.084824] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2059.747310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2059.747793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.749299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2059.758488] Lustre: Skipped 1 previous similar message [ 2064.865834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2064.868208] Lustre: Skipped 2 previous similar messages [ 2065.492728] Lustre: server umount lustre-OST0000 complete [ 2069.985221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.985659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2069.986350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2069.990424] Lustre: Skipped 1 previous similar message [ 2075.104754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2075.107538] Lustre: Skipped 2 previous similar messages [ 2080.224067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2080.226992] Lustre: Skipped 1 previous similar message [ 2081.247118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2081.292823] Lustre: server umount lustre-MDT0000 complete [ 2083.151576] Lustre: server umount lustre-MDT0001 complete [ 2084.828339] Lustre: server umount MGS complete [ 2090.207463] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:50:42 (1789368642) [ 2104.387879] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 2109.067489] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2126.736892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.667559] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2133.933292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.041775] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2134.064912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2134.123094] Lustre: lustre-MDT0001: new disk, initializing [ 2134.161938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2134.175228] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2134.179115] Lustre: 93269:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2134.183570] LustreError: 93269:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2136.046923] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2142.216584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2142.364796] Lustre: lustre-OST0000: new disk, initializing [ 2142.366614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2142.370556] Lustre: 94034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2142.425072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.476110] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2148.387427] Lustre: 94491:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000e9b8b8f1 x1876288948178432/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789368718 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2150.619612] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2154.053532] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.065319] Lustre: MGS: Client ee197847-3ce4-4c11-be99-b661d95dec65 (at 0@lo) reconnecting [ 2154.092459] Lustre: 92534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2154.101710] LustreError: 93255:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2154.108321] LustreError: 93255:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2154.117176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2154.186538] Lustre: lustre-MDT0000: new disk, initializing [ 2154.289947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.302868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2155.359767] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2155.364454] Lustre: 95104:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000fee114ad x1876288948184448/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789368725 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2156.076479] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2157.414899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2157.419882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2157.496577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2178.016904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.025746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2178.030938] Lustre: Skipped 1 previous similar message [ 2181.750684] Lustre: server umount lustre-OST0000 complete [ 2188.256555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2188.257535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.269733] Lustre: Skipped 2 previous similar messages [ 2190.068775] Lustre: server umount lustre-MDT0000 complete [ 2198.500280] Lustre: server umount lustre-MDT0001 complete [ 2200.438455] Lustre: server umount MGS complete [ 2206.134961] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:52:38 (1789368758) [ 2216.141568] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2217.919384] Key type lgssc unregistered [ 2218.123724] LNet: 97597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.132595] LNetError: 97597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.149515] LNet: Removed LNI 192.168.201.104@tcp [ 2218.668529] Key type .llcrypt unregistered [ 2218.669881] Key type ._llcrypt unregistered [ 2228.494119] Key type ._llcrypt registered [ 2228.495656] Key type .llcrypt registered [ 2228.549791] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2229.193049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2229.204569] alg: No test for adler32 (adler32-zlib) [ 2230.114881] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2230.230272] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2231.847203] Key type lgssc registered [ 2232.571691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2236.321123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2236.335526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.570564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.118433] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2244.431208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.623453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.663699] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2249.771257] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2252.214327] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2255.224816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2255.414480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.728190] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2262.404173] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2262.572124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2277.856531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2277.864137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.875077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.905571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.908438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.476723] Lustre: server umount lustre-OST0000 complete [ 2285.026362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.027458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2285.036235] Lustre: Skipped 1 previous similar message [ 2285.044278] Lustre: Skipped 2 previous similar messages [ 2290.145447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.148911] Lustre: Skipped 1 previous similar message [ 2295.267879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.274829] Lustre: Skipped 1 previous similar message [ 2299.359147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2299.417050] Lustre: server umount lustre-MDT0000 complete [ 2300.385320] LustreError: 101279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.398183] LustreError: 101279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2300.977451] LustreError: 98774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368855 with bad export cookie 2124294698601925021 [ 2300.980528] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.982918] LustreError: 98774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2301.144773] Lustre: server umount lustre-MDT0001 complete [ 2306.494382] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2307.964288] Key type lgssc unregistered [ 2308.153873] LNet: 102342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.158651] LNetError: 102342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.170205] LNet: Removed LNI 192.168.201.104@tcp [ 2308.626058] Key type .llcrypt unregistered [ 2308.628136] Key type ._llcrypt unregistered [ 2317.283375] Key type ._llcrypt registered [ 2317.284520] Key type .llcrypt registered [ 2317.336976] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2317.694541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.746852] alg: No test for adler32 (adler32-zlib) [ 2318.644469] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2318.752605] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2320.343216] Key type lgssc registered [ 2320.780287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2323.153659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.157451] LustreError: 103398:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2323.161958] LustreError: 103402:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2323.164973] LustreError: 103398:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2323.168074] LustreError: 103398:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2323.172294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.174324] LustreError: 103398:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2324.354401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.652663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.763765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2329.406245] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2331.442667] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing lsmod [ 2332.664176] LustreError: 104604:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2332.669400] LustreError: 104605:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2332.672338] LustreError: 104604:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 2332.676070] LustreError: 104604:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2332.682275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2332.682532] LustreError: 104604:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2332.847218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.041130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.042789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.045599] Lustre: Skipped 1 previous similar message [ 2334.053122] Lustre: Skipped 2 previous similar messages [ 2337.121284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.123893] Lustre: Skipped 2 previous similar messages [ 2340.188584] Lustre: server umount lustre-MDT0000 complete [ 2341.764479] LustreError: 103414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368895 with bad export cookie 17050584682209346288 [ 2341.766491] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.770106] LustreError: 103414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2341.776467] Lustre: 103431:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2342.368074] LustreError: 103435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.368262] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.376342] LustreError: 103435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2342.381381] Lustre: Skipped 1 previous similar message [ 2342.387439] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2343.265703] LustreError: 103869:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000046daaa4 x1876289254342528/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2343.362669] Lustre: server umount lustre-MDT0001 complete [ 2345.142624] Lustre: server umount lustre-OST0000 complete [ 2348.931473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.145168] LustreError: 105668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.183283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.186890] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2354.656246] LustreError: 105669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.811786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.984732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2356.828516] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2360.034390] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2362.636870] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2365.972149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2366.207344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.862038] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2372.170441] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2374.377425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 2389.985628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2389.992517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.006301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.011157] Lustre: Skipped 1 previous similar message [ 2395.108789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2395.113269] Lustre: Skipped 1 previous similar message [ 2400.223141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2400.297502] Lustre: server umount lustre-OST0000 complete [ 2405.344845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.347476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2405.350455] Lustre: Skipped 2 previous similar messages [ 2405.353474] Lustre: Skipped 2 previous similar messages [ 2408.138503] Lustre: server umount lustre-MDT0000 complete [ 2409.874504] LustreError: 105650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368963 with bad export cookie 17050584682209346967 [ 2409.876787] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.886625] LustreError: 105650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2410.072264] Lustre: server umount lustre-MDT0001 complete [ 2414.730610] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2416.140548] Key type lgssc unregistered [ 2416.297550] LNet: 109218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2416.302234] LNetError: 109218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2416.313513] LNet: Removed LNI 192.168.201.104@tcp [ 2416.694825] Key type .llcrypt unregistered [ 2416.696257] Key type ._llcrypt unregistered [ 2427.239756] Key type ._llcrypt registered [ 2427.241267] Key type .llcrypt registered [ 2427.284226] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2427.632260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2427.679398] alg: No test for adler32 (adler32-zlib) [ 2428.561407] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2428.673850] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2430.263203] Key type lgssc registered [ 2430.731771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2434.267145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2434.282734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.559913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.432817] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2441.738105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.922275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2443.755895] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2446.986453] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2449.446747] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2452.720498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2452.878277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.732968] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2459.103936] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2460.075112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2471.372977] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2472.662516] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2475.487885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2475.490891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.496650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.512979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.514852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.518892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.040769] Lustre: server umount lustre-OST0000 complete [ 2483.168371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2483.173757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.180513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.182760] Lustre: Skipped 1 previous similar message [ 2486.754370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.755087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.987020] Lustre: server umount lustre-MDT0000 complete [ 2489.857649] LustreError: 110592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369043 with bad export cookie 15865125068078722144 [ 2489.863513] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2489.863750] LustreError: 110592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2490.030616] Lustre: server umount lustre-MDT0001 complete [ 2494.802687] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2496.108339] Key type lgssc unregistered [ 2496.261425] LNet: 114263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.265601] LNetError: 114263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.278480] LNet: Removed LNI 192.168.201.104@tcp [ 2496.688786] Key type .llcrypt unregistered [ 2496.689803] Key type ._llcrypt unregistered [ 2505.030215] Key type ._llcrypt registered [ 2505.031809] Key type .llcrypt registered [ 2505.102697] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2505.539871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2505.561989] alg: No test for adler32 (adler32-zlib) [ 2506.459447] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2506.564882] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2508.176865] Key type lgssc registered [ 2508.587384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2511.970490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2511.987375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.196655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.021221] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2519.092758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.288686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2521.290946] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2524.601228] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.177564] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2530.700409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2530.886702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.049878] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2537.573774] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2538.088127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2548.704925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2548.711696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.724186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.729064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.729989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2549.737665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.574624] Lustre: server umount lustre-OST0000 complete [ 2555.360876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2555.364662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.371096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2555.373716] Lustre: Skipped 1 previous similar message [ 2559.968293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.971810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.976467] Lustre: Skipped 1 previous similar message [ 2561.582145] Lustre: server umount lustre-MDT0000 complete [ 2563.444679] LustreError: 117956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369117 with bad export cookie 9582136671923641674 [ 2563.453516] LustreError: 117956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2563.453712] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2563.614676] Lustre: server umount lustre-MDT0001 complete [ 2568.791385] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2570.152750] Key type lgssc unregistered [ 2570.306750] LNet: 118992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2570.309940] LNetError: 118992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2570.316768] LNet: Removed LNI 192.168.201.104@tcp [ 2570.724061] Key type .llcrypt unregistered [ 2570.727327] Key type ._llcrypt unregistered [ 2582.887557] Key type ._llcrypt registered [ 2582.888763] Key type .llcrypt registered [ 2582.946092] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2583.570728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2583.580574] alg: No test for adler32 (adler32-zlib) [ 2584.451793] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2584.570141] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2586.167234] Key type lgssc registered [ 2586.643628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2590.242781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2590.253918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.496202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.397882] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2597.090663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.255752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2598.886709] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2602.009893] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2604.420869] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2607.599225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2607.794989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.301242] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2613.852095] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2615.984167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 2628.457584] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2630.494378] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2630.497222] Lustre: Skipped 2 previous similar messages [ 2632.689638] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2632.693381] Lustre: Skipped 2 previous similar messages [ 2634.750319] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2634.764551] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2640.645787] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.4@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2640.656073] Lustre: Skipped 2 previous similar messages [ 2641.507845] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.4@tcp 192.168.0.[2,10]@tcp [ 2646.634602] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.4/24@tcp in log lustre-MDT0000 [ 2646.637220] Lustre: Skipped 2 previous similar messages [ 2648.386262] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.4/24@tcp [ 2648.391652] Lustre: Skipped 1 previous similar message [ 2657.248173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2657.251807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.258463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.273908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.274091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.279653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.835425] Lustre: server umount lustre-OST0000 complete [ 2663.393482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.395073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.403464] Lustre: Skipped 1 previous similar message [ 2663.414074] Lustre: Skipped 2 previous similar messages [ 2668.512611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.519711] Lustre: Skipped 1 previous similar message [ 2668.888149] Lustre: server umount lustre-MDT0000 complete [ 2670.896558] LustreError: 123031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369224 with bad export cookie 4932868026166802786 [ 2670.898659] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2670.904993] LustreError: 123031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2671.106106] Lustre: server umount lustre-MDT0001 complete [ 2676.638376] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2678.125831] Key type lgssc unregistered [ 2678.321917] LNet: 125138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2678.329625] LNetError: 125138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2678.337817] LNet: Removed LNI 192.168.201.104@tcp [ 2678.836822] Key type .llcrypt unregistered [ 2678.838192] Key type ._llcrypt unregistered [ 2693.049086] Key type ._llcrypt registered [ 2693.050828] Key type .llcrypt registered [ 2693.104988] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2693.628974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2693.669410] alg: No test for adler32 (adler32-zlib) [ 2694.575257] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2694.688488] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2696.311207] Key type lgssc registered [ 2696.852656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2700.801840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2700.816361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.004982] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2702.017406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.020201] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.4/24@tcp [ 2704.206905] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2708.606333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.799662] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2708.812970] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.4/24@tcp [ 2708.840460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2710.939450] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2714.250256] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2717.065866] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2720.611335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.756293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.478727] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2726.696207] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2727.924475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 2738.144332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2738.147496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.151173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2738.158145] Lustre: Skipped 1 previous similar message [ 2742.509345] Lustre: server umount lustre-OST0000 complete [ 2744.806629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2744.810492] LustreError: Skipped 1 previous similar message [ 2744.812821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2744.820371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.823369] Lustre: Skipped 1 previous similar message [ 2748.384374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2748.386506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.440196] Lustre: server umount lustre-MDT0000 complete [ 2751.896188] LustreError: 126652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369305 with bad export cookie 5443825540027374473 [ 2751.898604] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.901052] LustreError: 126652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2752.048841] Lustre: server umount lustre-MDT0001 complete [ 2756.736520] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2757.982267] Key type lgssc unregistered [ 2758.169426] LNet: 130268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2758.177619] LNetError: 130268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2758.189703] LNet: Removed LNI 192.168.201.104@tcp [ 2758.568447] Key type .llcrypt unregistered [ 2758.569958] Key type ._llcrypt unregistered [ 2769.267350] Key type ._llcrypt registered [ 2769.269053] Key type .llcrypt registered [ 2769.319540] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 2776.324455] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2776.726112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2776.774044] alg: No test for adler32 (adler32-zlib) [ 2777.648867] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2777.760379] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2779.359215] Key type lgssc registered [ 2779.789948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2792.922324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2792.927796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.009811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2794.022282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2794.063290] Lustre: lustre-MDT0000: new disk, initializing [ 2794.087392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.095988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2795.467587] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2800.075449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.124501] Lustre: 133635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2800.153513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2800.155477] Lustre: Skipped 1 previous similar message [ 2800.198136] Lustre: lustre-MDT0001: new disk, initializing [ 2800.242277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2800.256322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2800.261304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2801.902782] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2805.995560] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2808.245908] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2811.207835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.341975] Lustre: lustre-OST0000: new disk, initializing [ 2811.343963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2811.347886] Lustre: 135601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2811.403357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.785980] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2816.520867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2816.524846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2816.577636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2818.305435] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2819.971247] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2820.057699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2821.501272] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2821.579064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2826.720777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2826.721674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.723569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2826.724221] LustreError: Skipped 1 previous similar message [ 2826.730450] Lustre: Skipped 1 previous similar message [ 2828.382849] Lustre: server umount lustre-OST0000 complete [ 2831.327802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2831.330237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.334775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.336682] Lustre: Skipped 1 previous similar message [ 2836.960097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2836.962194] Lustre: Skipped 2 previous similar messages [ 2842.080102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.082246] Lustre: Skipped 1 previous similar message [ 2844.127127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2844.194442] Lustre: server umount lustre-MDT0000 complete [ 2845.422226] LustreError: 134431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369399 with bad export cookie 8872607689068983870 [ 2845.424520] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2845.426864] LustreError: 134431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2845.549571] Lustre: server umount lustre-MDT0001 complete [ 2848.274112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.431127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2849.731478] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2852.410881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.520124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2853.772619] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2856.107497] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2857.881947] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2860.166839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.154923] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2864.589901] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2866.108051] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2875.355424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2876.850304] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2876.936110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2880.366465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.438829] Lustre: lustre-OST0001: new disk, initializing [ 2880.440446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2880.442698] Lustre: 141251:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2880.471977] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2880.475300] Lustre: Skipped 1 previous similar message [ 2882.455717] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2885.725134] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2887.678211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2887.681504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2887.707705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2887.827101] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2889.171547] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2889.240840] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2891.045928] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89c0208d2800.ost_server_uuid 50 [ 2891.551404] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89c0208d2800.ost_server_uuid in FULL state after 0 sec [ 2897.888774] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2897.891758] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.892886] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2897.896029] Lustre: Skipped 2 previous similar messages [ 2899.518065] Lustre: server umount lustre-OST0001 complete [ 2903.009399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.009618] LustreError: 139634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.013752] Lustre: Skipped 1 previous similar message [ 2903.019060] LustreError: 139634:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2908.128781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2908.130367] LustreError: 139634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.130869] Lustre: Skipped 4 previous similar messages [ 2913.252662] LustreError: 139634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.257584] LustreError: 139634:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2915.295129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2915.347464] Lustre: server umount lustre-OST0000 complete [ 2918.368807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.373544] Lustre: Skipped 1 previous similar message [ 2922.711564] Lustre: server umount lustre-MDT0000 complete [ 2923.489225] LustreError: 137811:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.495659] LustreError: 137811:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2924.043286] LustreError: 137798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369478 with bad export cookie 8872607689068984738 [ 2924.046453] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2924.047445] LustreError: 137798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2924.173319] Lustre: server umount lustre-MDT0001 complete [ 2930.761425] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 2931.817202] Key type lgssc unregistered [ 2931.951663] LNet: 144267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2931.954652] LNetError: 144267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2931.967318] LNet: Removed LNI 192.168.201.104@tcp [ 2932.287330] Key type .llcrypt unregistered [ 2932.288602] Key type ._llcrypt unregistered [ 2947.143367] Key type ._llcrypt registered [ 2947.144575] Key type .llcrypt registered [ 2947.187561] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 2953.596251] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2953.916565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2953.923857] alg: No test for adler32 (adler32-zlib) [ 2954.792187] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 2954.884623] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 2956.479208] Key type lgssc registered [ 2956.899947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2968.941321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2968.950445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.039670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2970.049691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2970.090821] Lustre: lustre-MDT0000: new disk, initializing [ 2970.126474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.135237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2971.637482] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2976.013523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.057847] Lustre: 148388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2976.081365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2976.083821] Lustre: Skipped 1 previous similar message [ 2976.119825] Lustre: lustre-MDT0001: new disk, initializing [ 2976.176356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2976.189725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2976.193966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2977.566622] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2981.479584] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2983.299953] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2985.548572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.658274] Lustre: lustre-OST0000: new disk, initializing [ 2985.660656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2985.664369] Lustre: 150355:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2985.696401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.695338] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2988.408335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2988.412463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2988.452361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2991.550619] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2994.501114] Lustre: Failing over lustre-OST0000 [ 2994.558949] Lustre: server umount lustre-OST0000 complete [ 3007.417937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3007.526257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.529977] Lustre: Skipped 1 previous similar message [ 3007.537215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.543716] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3009.187085] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3009.529609] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3010.998354] Lustre: Failing over lustre-MDT0000 [ 3011.097205] Lustre: server umount lustre-MDT0000 complete [ 3012.063804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3012.067090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.111236] Lustre: 146613:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789369552/real 1789369552] req@00000000a74a1850 x1876289921248512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789369568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3014.623325] Lustre: 146615:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789369552/real 1789369552] req@000000006e25292b x1876289921248896/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789369568 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3017.696155] LustreError: 149582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.702120] LustreError: 149582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 3019.415633] LustreError: 148396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.679151] Lustre: 146616:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789369557/real 1789369557] req@00000000aba3ec21 x1876289921250048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789369573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3022.815914] LustreError: 148399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.821318] LustreError: 148399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3024.255165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.294905] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3024.300483] Lustre: 148391:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3024.371411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3024.388286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.414072] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3024.536253] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3025.746108] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3029.482145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3029.491959] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3029.513027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3034.594702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.596816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3034.596843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3034.598776] Lustre: Skipped 3 previous similar messages [ 3036.627671] Lustre: server umount lustre-OST0000 complete [ 3039.712634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.712836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3039.713698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3039.713703] Lustre: Skipped 1 previous similar message [ 3039.717493] Lustre: Skipped 1 previous similar message [ 3039.719930] LustreError: Skipped 1 previous similar message [ 3044.052850] Lustre: server umount lustre-MDT0000 complete [ 3044.832076] LustreError: 149581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3044.837838] LustreError: 149581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3045.423677] LustreError: 148381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369599 with bad export cookie 2732481544523309145 [ 3045.426487] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.428396] LustreError: 148381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3045.562834] Lustre: server umount lustre-MDT0001 complete [ 3049.593928] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 3050.696827] Key type lgssc unregistered [ 3050.833420] LNet: 154004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3050.836620] LNetError: 154004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3050.844632] LNet: Removed LNI 192.168.201.104@tcp [ 3051.182424] Key type .llcrypt unregistered [ 3051.184122] Key type ._llcrypt unregistered [ 3072.087758] Key type ._llcrypt registered [ 3072.089165] Key type .llcrypt registered [ 3072.143149] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3072.549920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3072.600985] alg: No test for adler32 (adler32-zlib) [ 3073.467332] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 3073.575198] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 3075.167172] Key type lgssc registered [ 3075.604195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3078.718866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3078.727745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.925955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.560944] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3085.794317] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3087.857367] LustreError: 155546:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3087.861251] LustreError: 155546:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3087.863671] LustreError: 155546:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 3087.978492] Lustre: server umount lustre-MDT0000 complete [ 3093.269277] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3096.617291] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3110.228618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.330199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3110.344849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3110.381975] Lustre: lustre-MDT0000: new disk, initializing [ 3110.422720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.432912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3112.016195] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3116.525816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.570615] Lustre: 158840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3116.595932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3116.598290] Lustre: Skipped 1 previous similar message [ 3116.634465] Lustre: lustre-MDT0001: new disk, initializing [ 3116.667019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3116.679244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3116.682634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3118.125816] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3122.203878] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3124.227715] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3126.984860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3127.096527] Lustre: lustre-OST0000: new disk, initializing [ 3127.098509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3127.101862] Lustre: 160809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3127.138344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.291159] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3133.430772] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3134.473666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3134.478404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3134.510591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3140.447159] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3141.013957] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:08:14 (1789369694) [ 3214.303420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3214.306388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.310563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3216.351666] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.351822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3216.360692] Lustre: Skipped 1 previous similar message [ 3216.363471] Lustre: Skipped 1 previous similar message [ 3221.471838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3221.476456] Lustre: Skipped 2 previous similar messages [ 3226.591730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3226.594859] Lustre: Skipped 2 previous similar messages [ 3228.127141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3228.201732] Lustre: server umount lustre-MDT0000 complete [ 3229.537548] LustreError: 159636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369783 with bad export cookie 8622414437713589298 [ 3229.538407] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3229.541817] LustreError: 159636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3231.711526] LustreError: 158847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.711620] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.717415] LustreError: 158847:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3231.722230] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3234.719489] LustreError: 160004:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.688605] Lustre: server umount lustre-MDT0001 complete [ 3243.211765] Lustre: server umount lustre-OST0000 complete [ 3246.657577] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3249.876634] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3262.582364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.665235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3262.676714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3262.713976] Lustre: lustre-MDT0000: new disk, initializing [ 3262.741540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.748938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3264.125930] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3268.507268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.545985] Lustre: 165441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3268.569271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3268.571741] Lustre: Skipped 1 previous similar message [ 3268.607924] Lustre: lustre-MDT0001: new disk, initializing [ 3268.654751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3268.658767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3270.172399] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3274.303453] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3276.287655] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3278.697778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3278.798364] Lustre: lustre-OST0000: new disk, initializing [ 3278.800549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3278.803836] Lustre: 167408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3278.836295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.839233] Lustre: Skipped 1 previous similar message [ 3280.946913] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3284.804868] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3285.502470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3285.506403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3285.547118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3286.295854] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3286.372570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3287.731031] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3287.813661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3290.594596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.594722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3290.595261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3290.595265] Lustre: Skipped 1 previous similar message [ 3290.598410] Lustre: Skipped 1 previous similar message [ 3290.600580] LustreError: Skipped 1 previous similar message [ 3294.542496] Lustre: server umount lustre-OST0000 complete [ 3299.295919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3299.298870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3310.047108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3310.048399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3310.052364] Lustre: Skipped 7 previous similar messages [ 3310.113621] Lustre: server umount lustre-MDT0000 complete [ 3311.385206] LustreError: 165431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369865 with bad export cookie 8622414437713748100 [ 3311.388198] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3311.390587] LustreError: 165431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3311.504356] Lustre: server umount lustre-MDT0001 complete [ 3314.239233] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:11:07 (1789369867) [ 3319.517457] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3322.803100] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3335.250971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.335467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3335.350655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3335.389139] Lustre: lustre-MDT0000: new disk, initializing [ 3335.417921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.424029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3336.835509] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3341.054214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.106505] Lustre: 172155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3341.206930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3342.721538] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3346.532414] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3348.507735] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3350.776640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3350.882520] Lustre: 174133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3353.018876] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3356.663559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3356.697141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3356.741169] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3361.760412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3361.761298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.763193] LustreError: Skipped 1 previous similar message [ 3361.767476] Lustre: Skipped 1 previous similar message [ 3361.768198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3361.772012] Lustre: Skipped 1 previous similar message [ 3364.691115] Lustre: server umount lustre-OST0000 complete [ 3366.880887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3380.191794] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3380.252344] Lustre: server umount lustre-MDT0000 complete [ 3381.217374] LustreError: 172168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.224232] LustreError: 172168:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3381.590490] LustreError: 172149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369935 with bad export cookie 8622414437713748982 [ 3381.592838] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3381.595061] LustreError: 172149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3381.724855] Lustre: server umount lustre-MDT0001 complete [ 3384.888856] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:12:17 (1789369937) [ 3390.454975] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3393.630254] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3405.417509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.495298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3405.507239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3405.508824] Lustre: Skipped 3 previous similar messages [ 3405.543161] Lustre: lustre-MDT0000: new disk, initializing [ 3405.544855] Lustre: Skipped 2 previous similar messages [ 3405.565461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.568261] Lustre: Skipped 2 previous similar messages [ 3405.574851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3405.578223] Lustre: Skipped 2 previous similar messages [ 3406.988550] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3411.338953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.383054] Lustre: 178446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3411.481789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3412.892875] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3416.718127] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3418.607029] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3420.909055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3421.020358] Lustre: 180424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3423.129250] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3427.075478] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3428.899812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3439.074518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3439.074666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.077471] LustreError: Skipped 1 previous similar message [ 3439.080827] Lustre: Skipped 3 previous similar messages [ 3439.081356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3439.085883] Lustre: Skipped 7 previous similar messages [ 3440.329243] Lustre: server umount lustre-OST0000 complete [ 3447.639455] Lustre: server umount lustre-MDT0000 complete [ 3448.824015] LustreError: 178440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370002 with bad export cookie 8622414437713750053 [ 3448.826093] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3448.827673] LustreError: 178440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3448.943555] Lustre: server umount lustre-MDT0001 complete [ 3452.438153] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 3453.489785] Key type lgssc unregistered [ 3453.626319] LNet: 182277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3453.629306] LNetError: 182277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3453.638302] LNet: Removed LNI 192.168.201.104@tcp [ 3453.931634] Key type .llcrypt unregistered [ 3453.932714] Key type ._llcrypt unregistered [ 3462.499582] Key type ._llcrypt registered [ 3462.501252] Key type .llcrypt registered [ 3462.539064] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3462.873722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3462.926304] alg: No test for adler32 (adler32-zlib) [ 3463.769205] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 3463.849866] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 3465.431240] Key type lgssc registered [ 3465.776781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3468.397965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3468.405222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.577234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.898706] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3473.708527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.825382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3475.140980] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3477.531120] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3479.236643] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3481.459230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.592518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3483.590189] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3486.048810] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3495.909975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3509.215823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3509.217494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.220809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3511.264781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3511.265939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3511.266204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.151283] Lustre: server umount lustre-OST0000 complete [ 3519.967786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3519.970026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.973867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.975498] Lustre: Skipped 1 previous similar message [ 3522.534570] Lustre: server umount lustre-MDT0000 complete [ 3523.864038] LustreError: 185507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370077 with bad export cookie 6564195684924778405 [ 3523.867817] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.867879] LustreError: 185507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3523.979625] Lustre: server umount lustre-MDT0001 complete [ 3527.163882] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 3528.121363] Key type lgssc unregistered [ 3528.243400] LNet: 187217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3528.246272] LNetError: 187217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3528.252303] LNet: Removed LNI 192.168.201.104@tcp [ 3528.558267] Key type .llcrypt unregistered [ 3528.559182] Key type ._llcrypt unregistered [ 3538.173188] Key type ._llcrypt registered [ 3538.174456] Key type .llcrypt registered [ 3538.215445] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3538.481867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3538.493590] alg: No test for adler32 (adler32-zlib) [ 3539.350646] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 3539.448815] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 3541.031202] Key type lgssc registered [ 3541.434399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3544.240071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3544.249156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.427599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.768705] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3549.394915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.517477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3550.757641] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3552.972966] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3554.668404] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3556.805817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.909616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.652663] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3560.853093] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3570.210498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3575.265632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3575.265957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.268340] LustreError: Skipped 1 previous similar message [ 3575.268899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3575.271511] Lustre: Skipped 1 previous similar message [ 3576.478256] Lustre: lustre-OST0000: Not available for connect from 192.168.201.4@tcp (stopping) [ 3576.480331] Lustre: Skipped 1 previous similar message [ 3579.595334] Lustre: server umount lustre-OST0000 complete [ 3581.874667] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89c018d07000.ost_server_uuid 50 [ 3582.354905] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3592.321885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.322554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3592.326425] Lustre: Skipped 1 previous similar message [ 3592.330535] Lustre: Skipped 1 previous similar message [ 3595.424349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.426441] Lustre: Skipped 1 previous similar message [ 3598.485331] Lustre: server umount lustre-MDT0000 complete [ 3599.682217] LustreError: 188598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370153 with bad export cookie 4357965133590991618 [ 3599.685757] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3599.685817] LustreError: 188598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3599.799040] Lustre: server umount lustre-MDT0001 complete [ 3602.519345] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:15:55 (1789370155) [ 3604.585164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.715808] LustreError: 192526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3604.721050] LustreError: 192526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3604.736333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.035357] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3608.798124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.911043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3610.205815] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3612.505469] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3614.306366] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3616.533081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3618.590338] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3620.898747] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3623.008322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3623.075866] Lustre: lustre-OST0001: new disk, initializing [ 3623.077722] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3623.080164] Lustre: 195365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3623.108219] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3623.110136] Lustre: Skipped 1 previous similar message [ 3624.901307] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3628.405296] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3629.289975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3631.615079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3631.618180] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3631.642681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3634.656860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3634.660452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.664637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3636.708166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.634293] Lustre: server umount lustre-OST0000 complete [ 3641.826333] LustreError: 196087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.831814] LustreError: 196087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3642.498558] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3642.566728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3643.798121] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3643.868462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3644.056765] LustreError: 194346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.945616] LustreError: 194347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.950742] LustreError: 194347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3652.067263] LustreError: 194344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.072114] LustreError: 194344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3657.185739] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3657.187110] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.187890] LustreError: Skipped 1 previous similar message [ 3657.191777] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3657.194100] Lustre: Skipped 4 previous similar messages [ 3659.974204] Lustre: server umount lustre-OST0001 complete [ 3667.226145] Lustre: server umount lustre-MDT0000 complete [ 3667.424095] LustreError: 192527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.430985] LustreError: 192527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3668.470202] LustreError: 195364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370222 with bad export cookie 4357965133590992675 [ 3668.473653] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.475980] LustreError: 195364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3668.590794] Lustre: server umount lustre-MDT0001 complete [ 3676.236200] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:17:09 (1789370229) [ 3678.440806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.545526] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3678.560936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3678.633945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.010294] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3682.793518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.826846] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3682.841496] Lustre: 198779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3684.362130] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3686.806556] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3688.646235] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3690.946173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3691.016096] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3692.920570] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3694.414712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3695.148285] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3697.259231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3697.283310] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3699.088222] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3701.254529] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3704.799993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3704.800241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.802386] LustreError: Skipped 1 previous similar message [ 3704.802817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3704.808445] Lustre: Skipped 4 previous similar messages [ 3704.809735] Lustre: Skipped 2 previous similar messages [ 3708.094149] Lustre: server umount lustre-OST0000 complete [ 3710.440731] LustreError: 200624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.445484] LustreError: 200624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3720.673423] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3720.675790] LustreError: Skipped 1 previous similar message [ 3724.808570] Lustre: server umount lustre-OST0001 complete [ 3729.375924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3729.379973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.384054] Lustre: Skipped 2 previous similar messages [ 3732.191203] Lustre: server umount lustre-MDT0000 complete [ 3733.484240] LustreError: 198771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370287 with bad export cookie 4357965133590994061 [ 3733.486952] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3733.487968] LustreError: 198771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3733.607699] Lustre: server umount lustre-MDT0001 complete [ 3741.363588] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:18:14 (1789370294) [ 3746.837165] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3750.140691] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3762.140600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.235147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3762.247566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3762.289856] Lustre: lustre-MDT0000: new disk, initializing [ 3762.320055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.323201] Lustre: Skipped 3 previous similar messages [ 3762.329675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3763.654693] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3767.676048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.715532] Lustre: 206840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3767.735672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3767.737556] Lustre: Skipped 1 previous similar message [ 3767.770557] Lustre: lustre-MDT0001: new disk, initializing [ 3767.808308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3767.812107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3769.038469] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3772.602424] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3774.394757] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3776.572956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3776.667720] Lustre: lustre-OST0000: new disk, initializing [ 3776.669446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3776.671950] Lustre: 208804:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3778.566759] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3779.954639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3779.957899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3779.986836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3782.074069] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3783.395358] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3783.480415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3784.753458] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3784.820390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3787.469649] Lustre: server umount lustre-OST0000 complete [ 3790.304746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.305271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3790.308232] Lustre: Skipped 2 previous similar messages [ 3790.309920] Lustre: Skipped 9 previous similar messages [ 3798.518688] Lustre: server umount lustre-MDT0000 complete [ 3799.671096] LustreError: 206832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370353 with bad export cookie 4357965133590995608 [ 3799.673405] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3799.675414] LustreError: 206832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3809.302207] Lustre: server umount lustre-MDT0001 complete [ 3812.098061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3812.240238] LustreError: 211018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.248086] LustreError: 211018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3813.610426] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3816.360669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.712338] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3819.861656] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3821.462214] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3823.498692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.304259] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3827.479038] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3828.761928] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3837.999534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3839.489389] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3839.583390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3842.017909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3846.350549] Lustre: server umount lustre-OST0000 complete [ 3848.215244] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3849.606024] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3873.197619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3873.312978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.317095] Lustre: Skipped 5 previous similar messages [ 3875.234215] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3877.541921] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3878.043947] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3881.100985] Lustre: server umount lustre-OST0000 complete [ 3882.282667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3882.283362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3882.287373] Lustre: Skipped 3 previous similar messages [ 3882.289146] Lustre: Skipped 6 previous similar messages [ 3884.374835] Lustre: server umount lustre-MDT0000 complete [ 3885.597626] LustreError: 211000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370439 with bad export cookie 4357965133590996476 [ 3885.600798] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3885.601826] LustreError: 211000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3885.712186] Lustre: server umount lustre-MDT0001 complete [ 3888.559482] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:20:41 (1789370441) [ 3890.772963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.925162] LustreError: 217038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3892.307572] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3895.104912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.595557] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3898.966267] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3900.719183] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3902.894369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.812726] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3907.126756] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3908.472269] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3915.671524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3917.115778] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3917.192157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3919.426601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3919.501471] Lustre: lustre-OST0001: new disk, initializing [ 3919.503185] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3919.505399] Lustre: 220457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3921.466454] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3925.301331] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3926.798337] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3928.576204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3928.579565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3928.606721] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3928.918448] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3930.343407] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3930.421963] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3933.667914] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.670566] LustreError: Skipped 2 previous similar messages [ 3937.152191] Lustre: server umount lustre-OST0001 complete [ 3939.122219] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3939.208495] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3940.622422] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3940.695442] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3964.341032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.303549] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3968.709215] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3969.307538] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3969.510503] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3969.514638] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3969.514699] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3969.522288] Lustre: Skipped 1 previous similar message [ 3976.060618] Lustre: server umount lustre-OST0001 complete [ 3994.195051] Lustre: server umount lustre-OST0000 complete [ 4001.748934] Lustre: server umount lustre-MDT0000 complete [ 4002.981156] LustreError: 217019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370557 with bad export cookie 4357965133590997407 [ 4002.984293] LustreError: 217019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4002.984314] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.136035] Lustre: server umount lustre-MDT0001 complete [ 4017.035487] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:22:50 (1789370570) [ 4019.207794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.304377] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4019.313871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4019.390997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.393435] Lustre: Skipped 5 previous similar messages [ 4020.685746] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4023.695095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.731086] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4023.741552] Lustre: 225703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4025.369416] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4027.882638] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4029.893584] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4032.263226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4032.327912] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4034.302221] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4036.729688] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4045.613133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4045.638527] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4047.857658] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4050.311813] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4051.883618] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 4053.464982] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4053.550368] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4055.142806] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4055.221162] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4057.100611] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89c021b70000.ost_server_uuid 50 [ 4057.673085] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89c021b70000.ost_server_uuid in FULL state after 0 sec [ 4058.034499] Lustre: Permanently deactivating lustre-OST0001 [ 4058.039789] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4058.909682] Lustre: Permanently reactivating lustre-OST0001 [ 4063.202286] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4063.204458] LustreError: Skipped 3 previous similar messages [ 4063.204573] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.206068] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4063.208755] Lustre: Skipped 9 previous similar messages [ 4063.210360] Lustre: Skipped 10 previous similar messages [ 4065.978461] Lustre: server umount lustre-OST0001 complete [ 4068.321610] LustreError: 227547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.326112] LustreError: 227547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 23 previous similar messages [ 4073.422059] Lustre: server umount lustre-OST0000 complete [ 4089.311118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4089.378108] Lustre: server umount lustre-MDT0000 complete [ 4090.573075] LustreError: 225694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370644 with bad export cookie 4357965133590998681 [ 4090.576549] LustreError: 225694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4090.577016] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4090.691720] Lustre: server umount lustre-MDT0001 complete [ 4098.352895] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:24:11 (1789370651) [ 4101.464831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.589153] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4101.602985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4101.606374] Lustre: Skipped 5 previous similar messages [ 4103.855913] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4108.754530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.811750] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4108.830470] Lustre: 232381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4111.661186] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4115.875370] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4119.240663] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4122.835244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4122.985474] Lustre: Permanently deactivating lustre-OST0000 [ 4122.993832] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4123.788736] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4126.456794] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4130.223580] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4133.745887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4137.043793] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4139.603738] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4140.754832] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4141.584685] Lustre: Permanently reactivating lustre-OST0000 [ 4141.594488] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4141.597968] Lustre: Skipped 2 previous similar messages [ 4149.260753] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4149.265239] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4149.273310] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4149.294175] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4149.304225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4160.851766] Lustre: server umount lustre-OST0001 complete [ 4168.793329] Lustre: server umount lustre-OST0000 complete [ 4176.561237] Lustre: server umount lustre-MDT0000 complete [ 4178.645468] LustreError: 232373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370732 with bad export cookie 4357965133591000319 [ 4178.647296] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4178.650944] LustreError: 232373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4178.789038] Lustre: server umount lustre-MDT0001 complete [ 4182.938731] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4184.155558] Key type lgssc unregistered [ 4184.300608] LNet: 237255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4184.303436] LNetError: 237255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4184.314960] LNet: Removed LNI 192.168.201.104@tcp [ 4184.705315] Key type .llcrypt unregistered [ 4184.706424] Key type ._llcrypt unregistered [ 4195.026380] Key type ._llcrypt registered [ 4195.027519] Key type .llcrypt registered [ 4195.067630] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4195.561754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4195.585186] alg: No test for adler32 (adler32-zlib) [ 4196.483602] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 4196.608686] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4198.199218] Key type lgssc registered [ 4198.668793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4203.517612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4203.526399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4204.702060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.175318] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4209.518394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.575152] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4209.586276] Lustre: 238691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4209.591050] Lustre: Permanently deactivating lustre-MDT0001 [ 4209.596228] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4209.723834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4211.327451] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4212.465667] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4212.471193] LustreError: 238735:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4212.476108] LustreError: 238735:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4212.480717] LustreError: 238735:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 4214.187861] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4216.190282] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4218.999575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4219.187305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4223.658157] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4229.439304] LustreError: 240543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.450342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4229.458157] LustreError: 240543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4231.132035] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4234.601188] LustreError: 240542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.631132] LustreError: 240542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4234.636481] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4238.264206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4238.414416] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4243.453184] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4243.478507] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4243.838565] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4250.035712] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4253.620558] Lustre: Permanently reactivating lustre-MDT0001 [ 4253.649316] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4253.658274] Lustre: Skipped 2 previous similar messages [ 4259.008195] Lustre: lustre-MDT0001: Client 97966df3-8c23-4f44-9265-3e14dd74362f (at 192.168.201.4@tcp) reconnecting [ 4264.753170] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4266.254341] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4271.173493] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4271.346815] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4276.137526] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4276.552095] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4279.956475] Lustre: Permanently deactivating lustre-MDT0001 [ 4279.972401] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4279.976321] Lustre: Skipped 2 previous similar messages [ 4288.672737] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4296.674822] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4296.681272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4296.684724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4296.695594] Lustre: Skipped 1 previous similar message [ 4301.794984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.799020] Lustre: Skipped 6 previous similar messages [ 4302.936400] Lustre: server umount lustre-MDT0000 complete [ 4305.472421] LustreError: 238685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370859 with bad export cookie 3419302362967711648 [ 4305.474625] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4305.483044] LustreError: 238685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4305.759679] Lustre: server umount lustre-MDT0001 complete [ 4314.640168] Lustre: server umount lustre-OST0000 complete [ 4322.754760] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:27:55 (1789370875) [ 4330.089868] Lustre: server umount lustre-OST0001 complete [ 4334.494362] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 4341.190959] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4364.074805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.231916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4364.235854] Lustre: Skipped 2 previous similar messages [ 4364.258477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4364.351662] Lustre: lustre-MDT0000: new disk, initializing [ 4364.400453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4364.410483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4367.626493] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4375.922567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4376.060705] Lustre: 246999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4376.152414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4376.158894] Lustre: Skipped 1 previous similar message [ 4376.227816] Lustre: lustre-MDT0001: new disk, initializing [ 4376.306398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4376.338197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4376.355217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4379.265450] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4385.903931] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4389.988644] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4394.977736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4395.212279] Lustre: lustre-OST0000: new disk, initializing [ 4395.214495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4395.218956] Lustre: 248970:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4395.282494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.721505] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4404.768053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4404.773908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4404.875042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4406.513307] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4408.002972] LustreError: 249395:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4414.700425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4414.843581] Lustre: lustre-OST0001: new disk, initializing [ 4414.846482] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4414.851161] Lustre: 250082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4414.854345] Lustre: lustre-OST0001: Not available for connect from 192.168.201.4@tcp (not set up) [ 4418.057575] LustreError: 249395:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4418.072830] LustreError: 247007:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4419.159886] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4424.766409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4424.774166] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4424.876270] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4426.157395] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4427.250722] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4428.119150] LustreError: 247007:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4430.305047] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4430.313122] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.322650] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4430.325976] Lustre: Skipped 3 previous similar messages [ 4430.340730] Lustre: Skipped 1 previous similar message [ 4431.837927] Lustre: server umount lustre-OST0001 complete [ 4435.444224] LustreError: 248951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4435.460303] LustreError: 248951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4440.553865] LustreError: 248950:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.572865] LustreError: 248950:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4445.675140] LustreError: 248949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4445.681048] LustreError: 248949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4450.789762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.790250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4450.810940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4450.815800] LustreError: Skipped 1 previous similar message [ 4450.827537] Lustre: Skipped 2 previous similar messages [ 4453.281587] Lustre: server umount lustre-OST0000 complete [ 4458.464435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4458.472676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.485454] Lustre: Skipped 1 previous similar message [ 4461.028029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.030427] Lustre: Skipped 1 previous similar message [ 4464.595473] Lustre: server umount lustre-MDT0000 complete [ 4466.155242] LustreError: 247010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.180082] LustreError: 247010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4467.500648] LustreError: 246991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371021 with bad export cookie 3419302362967714455 [ 4467.500897] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4467.505347] LustreError: 246991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4473.879620] Lustre: server umount lustre-MDT0001 complete [ 4482.861697] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4484.972293] Key type lgssc unregistered [ 4485.221279] LNet: 252114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4485.228631] LNetError: 252114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4485.246189] LNet: Removed LNI 192.168.201.104@tcp [ 4485.977706] Key type .llcrypt unregistered [ 4485.980056] Key type ._llcrypt unregistered [ 4513.109223] Key type ._llcrypt registered [ 4513.111682] Key type .llcrypt registered [ 4513.204677] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4513.735350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4513.797141] alg: No test for adler32 (adler32-zlib) [ 4514.753632] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 4514.910336] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4516.527179] Key type lgssc registered [ 4517.386194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4522.642478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4522.659994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.817270] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4523.835507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4523.942248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.441606] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4531.381494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.483945] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4531.529709] Lustre: 254303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4531.711657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4534.600548] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4540.068187] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4544.057510] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4548.611147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4548.767031] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4548.901896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4552.723278] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4556.363305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4557.013818] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4602.336382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4602.341149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.345039] LustreError: Skipped 1 previous similar message [ 4602.348515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4602.354859] Lustre: Skipped 1 previous similar message [ 4606.997434] Lustre: server umount lustre-OST0000 complete [ 4609.900203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4618.172385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4618.359689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.444582] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4622.457637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4622.462506] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4622.472928] Lustre: Skipped 1 previous similar message [ 4622.739708] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4627.230587] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4663.782937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4663.783499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.787768] LustreError: Skipped 1 previous similar message [ 4663.789015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4663.789020] Lustre: Skipped 1 previous similar message [ 4663.810842] Lustre: Skipped 1 previous similar message [ 4668.900072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4668.905694] Lustre: Skipped 2 previous similar messages [ 4669.075984] Lustre: server umount lustre-OST0000 complete [ 4674.019202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4674.029535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4674.036143] Lustre: Skipped 1 previous similar message [ 4677.425275] Lustre: server umount lustre-MDT0000 complete [ 4679.501434] LustreError: 254295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371233 with bad export cookie 12902101370814728598 [ 4679.508066] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4679.512224] LustreError: 254295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4679.758344] Lustre: server umount lustre-MDT0001 complete [ 4686.709833] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4688.799343] Key type lgssc unregistered [ 4688.997964] LNet: 259717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4689.004335] LNetError: 259717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4689.015984] LNet: Removed LNI 192.168.201.104@tcp [ 4689.560964] Key type .llcrypt unregistered [ 4689.565486] Key type ._llcrypt unregistered [ 4704.143902] Key type ._llcrypt registered [ 4704.146522] Key type .llcrypt registered [ 4704.210532] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4704.815201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4704.949162] alg: No test for adler32 (adler32-zlib) [ 4705.873309] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 4706.040213] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4707.695184] Key type lgssc registered [ 4708.238560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4712.007798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4712.024187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.232891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.009786] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4719.399493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4719.598723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4721.765226] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4725.355440] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4727.947777] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4731.139301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.365262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4734.535593] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4737.775923] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4738.536078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4753.888123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4753.891296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.898017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4754.913949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4754.917300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4754.921265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4759.706231] Lustre: server umount lustre-OST0000 complete [ 4765.152727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.153286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4765.158739] Lustre: Skipped 1 previous similar message [ 4765.161946] Lustre: Skipped 1 previous similar message [ 4767.331842] Lustre: server umount lustre-MDT0000 complete [ 4768.671473] LustreError: 261088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371322 with bad export cookie 17369317158705011467 [ 4768.674373] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4768.676589] LustreError: 261088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4768.807685] Lustre: server umount lustre-MDT0001 complete [ 4772.675635] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4773.845683] Key type lgssc unregistered [ 4773.989844] LNet: 265406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4773.993141] LNetError: 265406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4774.000461] LNet: Removed LNI 192.168.201.104@tcp [ 4774.357776] Key type .llcrypt unregistered [ 4774.359377] Key type ._llcrypt unregistered [ 4782.155613] Key type ._llcrypt registered [ 4782.156939] Key type .llcrypt registered [ 4782.209779] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4782.599184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4782.629134] alg: No test for adler32 (adler32-zlib) [ 4783.501556] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 4783.598770] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4785.191234] Key type lgssc registered [ 4785.642158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4788.899485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4788.910129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.120880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.889598] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4795.608292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.774688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4797.557515] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4801.084764] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4803.593150] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4806.637387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4806.832695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.240589] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4812.563701] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4815.015487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4830.178208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4830.182823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.188847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4830.688633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4830.692986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.702219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4833.172049] Lustre: server umount lustre-OST0000 complete [ 4835.810555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4835.815884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4835.816644] Lustre: Skipped 1 previous similar message [ 4835.820245] Lustre: Skipped 1 previous similar message [ 4840.737379] Lustre: server umount lustre-MDT0000 complete [ 4840.928030] LustreError: 266615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4840.935249] LustreError: 266615:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4842.288340] LustreError: 266596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371396 with bad export cookie 8405311696034762388 [ 4842.292224] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4842.293296] LustreError: 266596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4842.415549] Lustre: server umount lustre-MDT0001 complete [ 4846.481426] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4847.640500] Key type lgssc unregistered [ 4847.777353] LNet: 270306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4847.781050] LNetError: 270306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4847.793424] LNet: Removed LNI 192.168.201.104@tcp [ 4848.142927] Key type .llcrypt unregistered [ 4848.144423] Key type ._llcrypt unregistered [ 4859.177162] Key type ._llcrypt registered [ 4859.178811] Key type .llcrypt registered [ 4859.240539] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4859.689869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4859.697311] alg: No test for adler32 (adler32-zlib) [ 4860.590458] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 4860.695277] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4862.287931] Key type lgssc registered [ 4862.773100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4866.581383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4866.593809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.809544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.946272] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4877.292278] LustreError: 271697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4877.303562] LustreError: 271697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4877.862323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4878.238630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4888.227692] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4905.348322] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4921.685962] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4935.834213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4937.066634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.180204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4946.941864] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4960.094024] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4989.409896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.417245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4989.442766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4989.465658] Lustre: Skipped 1 previous similar message [ 4991.712917] Lustre: server umount lustre-OST0000 complete [ 4996.577331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4996.594461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.609017] Lustre: Skipped 1 previous similar message [ 4996.622039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4999.651411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.665186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4999.695842] Lustre: Skipped 1 previous similar message [ 5001.700158] Lustre: server umount lustre-MDT0000 complete [ 5004.770747] LustreError: 271692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5004.791804] LustreError: 271692:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5005.083124] LustreError: 271678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371559 with bad export cookie 13691566556282260890 [ 5005.090529] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5005.104118] LustreError: 271678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5005.429693] Lustre: server umount lustre-MDT0001 complete [ 5015.917928] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 5018.526920] Key type lgssc unregistered [ 5018.773230] LNet: 276005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5018.778437] LNetError: 276005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5018.797302] LNet: Removed LNI 192.168.201.104@tcp [ 5019.508908] Key type .llcrypt unregistered [ 5019.511525] Key type ._llcrypt unregistered [ 5035.819875] Key type ._llcrypt registered [ 5035.822422] Key type .llcrypt registered [ 5035.942216] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5036.843894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5036.899827] alg: No test for adler32 (adler32-zlib) [ 5038.108803] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 5038.338396] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 5039.983314] Key type lgssc registered [ 5041.003695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.815736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5047.837810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5049.243321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5053.083597] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5059.565634] LustreError: 277244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.577465] LustreError: 277244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5061.692697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.986776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5066.201682] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5073.078136] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5078.446556] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5085.680511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5086.117742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5091.782856] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5094.337802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5099.559893] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5112.804159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5112.812894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.823993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5114.849551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.852317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5114.856805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.397660] Lustre: server umount lustre-OST0000 complete [ 5119.981809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.989923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5119.995662] Lustre: Skipped 1 previous similar message [ 5120.008996] Lustre: Skipped 1 previous similar message [ 5125.091023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5125.098024] Lustre: Skipped 2 previous similar messages [ 5126.006453] Lustre: server umount lustre-MDT0000 complete [ 5129.959269] LustreError: 277211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371683 with bad export cookie 8175760507193013294 [ 5129.962782] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5129.971851] LustreError: 277211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5130.308267] Lustre: server umount lustre-MDT0001 complete [ 5140.239899] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 5142.541197] Key type lgssc unregistered [ 5142.866196] LNet: 280943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5142.873821] LNetError: 280943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5142.903338] LNet: Removed LNI 192.168.201.104@tcp [ 5143.820773] Key type .llcrypt unregistered [ 5143.822707] Key type ._llcrypt unregistered [ 5158.280896] Key type ._llcrypt registered [ 5158.287552] Key type .llcrypt registered [ 5158.546922] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5184.116698] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 5199.826056] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5200.824354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5200.861884] alg: No test for adler32 (adler32-zlib) [ 5202.072306] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 5202.352485] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 5204.031317] Key type lgssc registered [ 5205.166145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5244.860122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5244.944834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5246.221408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5246.250598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5246.342550] Lustre: lustre-MDT0000: new disk, initializing [ 5246.416504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5246.433378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5250.804819] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5262.202223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5262.356699] Lustre: 284498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5262.402925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5262.405396] Lustre: Skipped 1 previous similar message [ 5262.461057] Lustre: lustre-MDT0001: new disk, initializing [ 5262.546932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5262.612860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5262.622089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5266.607436] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5276.643345] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5282.530739] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5289.675934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5290.049450] Lustre: lustre-OST0000: new disk, initializing [ 5290.055773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5290.064650] Lustre: 286466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5290.179495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5296.428738] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5296.707483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5296.724877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5296.865182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5307.127250] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5312.462085] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5312.637162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5317.217618] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5317.432754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5322.720428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5322.729400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.732693] LustreError: Skipped 1 previous similar message [ 5322.733659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5322.759922] Lustre: Skipped 1 previous similar message [ 5325.543298] Lustre: server umount lustre-OST0000 complete [ 5332.961176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.974534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5332.980198] Lustre: Skipped 1 previous similar message [ 5332.991943] Lustre: Skipped 1 previous similar message [ 5338.083330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5338.091138] Lustre: Skipped 2 previous similar messages [ 5342.179837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5342.189367] Lustre: Skipped 1 previous similar message [ 5344.223176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5344.475705] Lustre: server umount lustre-MDT0000 complete [ 5348.723759] LustreError: 285291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371902 with bad export cookie 864540586843040609 [ 5348.730448] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5348.734076] LustreError: 285291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5349.159474] Lustre: server umount lustre-MDT0001 complete [ 5357.780499] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:45:09 (1789371909) [ 5364.252857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.585968] LustreError: 288877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.607238] LustreError: 288877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5364.645779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5368.661802] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5369.834522] LustreError: 288878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.945862] LustreError: 288877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.370824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.795377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5382.256104] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5391.547187] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5399.282983] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5407.613700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5408.147336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.265605] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5423.910593] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.848293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5434.855900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.876841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5437.920229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5437.927065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.165510] Lustre: server umount lustre-OST0000 complete [ 5444.578867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5444.589730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.619212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5444.626626] Lustre: Skipped 2 previous similar messages [ 5458.400638] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5458.689499] Lustre: server umount lustre-MDT0000 complete [ 5462.608760] LustreError: 288859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372016 with bad export cookie 864540586843041491 [ 5462.623452] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.271351] Lustre: server umount lustre-MDT0001 complete [ 5475.877157] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 5478.941736] Key type lgssc unregistered [ 5479.239639] LNet: 292412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5479.246657] LNetError: 292412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5479.262258] LNet: Removed LNI 192.168.201.104@tcp [ 5480.103262] Key type .llcrypt unregistered [ 5480.105897] Key type ._llcrypt unregistered [ 5509.284851] Key type ._llcrypt registered [ 5509.286341] Key type .llcrypt registered [ 5509.420208] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 5525.719286] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5527.239970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5527.287379] alg: No test for adler32 (adler32-zlib) [ 5528.346302] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 5528.605972] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 5530.311250] Key type lgssc registered [ 5531.529698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5581.556609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5581.571215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.945862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5583.004898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5583.092226] Lustre: lustre-MDT0000: new disk, initializing [ 5583.184078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5583.229859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5588.043419] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5600.749710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.935398] Lustre: 296237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5601.004668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5601.009739] Lustre: Skipped 1 previous similar message [ 5601.114220] Lustre: lustre-MDT0001: new disk, initializing [ 5601.277461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5601.353891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5601.370826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5606.974496] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5616.244819] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5624.513978] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5635.672842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5636.176498] Lustre: lustre-OST2710: new disk, initializing [ 5636.191952] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5636.202106] Lustre: 298207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5636.352654] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5642.316090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5642.339319] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5642.586702] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5647.281534] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5661.466263] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5670.385951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5670.663916] Lustre: lustre-OST03e8: new disk, initializing [ 5670.667898] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5670.681984] Lustre: 299281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5670.791702] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5678.226944] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5680.821453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5680.836881] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5680.984786] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5685.707735] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5694.222350] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5694.540378] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5700.669120] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5700.878653] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5706.450093] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5706.701531] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5712.214929] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5712.534781] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5719.017406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5719.030482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.057895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5722.084141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.100938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5722.123288] Lustre: Skipped 3 previous similar messages [ 5724.806287] Lustre: server umount lustre-MDT0000 complete [ 5729.570515] LustreError: 296230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372283 with bad export cookie 16699451639384492650 [ 5729.582134] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5729.589340] LustreError: 296230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5730.461926] Lustre: server umount lustre-MDT0001 complete [ 5745.628283] Lustre: server umount lustre-OST2710 complete [ 5748.703367] Lustre: 294035:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789372286/real 1789372286] req@0000000072c00f8e x1876292620330240/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789372302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5748.733788] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5748.751801] Lustre: Skipped 2 previous similar messages [ 5749.755514] Lustre: server umount lustre-OST03e8 complete [ 5768.941914] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 5779.184584] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5821.887394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.129254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5822.153223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5822.229436] Lustre: lustre-MDT0000: new disk, initializing [ 5822.313835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.345744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5827.748707] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5841.066543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5841.309507] Lustre: 304535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5841.476332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5841.481762] Lustre: Skipped 1 previous similar message [ 5841.602300] Lustre: lustre-MDT0001: new disk, initializing [ 5841.796303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5841.890325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5841.918603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5848.294737] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5858.980909] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5865.277914] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5872.450958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5872.966393] Lustre: lustre-OST0000: new disk, initializing [ 5872.972649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5872.977024] Lustre: 306503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5873.106877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.378615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5878.398597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5878.603482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5880.538310] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5891.393631] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5896.199318] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5896.465907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5901.565487] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5901.813163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5904.352548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5904.356601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.364217] LustreError: Skipped 1 previous similar message [ 5904.369629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5904.408994] Lustre: Skipped 1 previous similar message [ 5909.503288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5909.514543] Lustre: Skipped 2 previous similar messages [ 5914.594499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5914.600195] Lustre: Skipped 1 previous similar message [ 5918.175402] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5918.441828] Lustre: server umount lustre-OST0000 complete [ 5923.809143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5923.816152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5923.826059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5928.850895] Lustre: server umount lustre-MDT0000 complete [ 5929.952900] LustreError: 304545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.971964] LustreError: 304545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5933.282901] LustreError: 304525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372487 with bad export cookie 16699451639384494344 [ 5933.294405] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5933.297697] LustreError: 304525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5933.798433] Lustre: server umount lustre-MDT0001 complete [ 5942.367695] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:54:54 (1789372494) [ 5943.816177] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5945.502800] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:54:57 (1789372497) [ 5950.134474] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5979.605073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.924605] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5979.949596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5980.126811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5985.950566] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5996.673399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.818055] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5996.856968] Lustre: 310362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6002.221920] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6024.494149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6024.765536] Lustre: Denying initial registration attempt from nid 192.168.201.104@tcp, specified as failover [ 6024.771290] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6024.779810] LustreError: 311816:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 6024.785376] LustreError: 311816:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6024.792270] LustreError: 311816:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6024.870703] Lustre: server umount lustre-OST0000 complete [ 6024.877329] LustreError: 311816:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6032.865040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6032.873098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.883426] Lustre: Skipped 1 previous similar message [ 6032.890827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6032.906713] Lustre: Skipped 2 previous similar messages [ 6038.489563] Lustre: server umount lustre-MDT0000 complete [ 6038.528701] LustreError: 310374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.559905] LustreError: 310374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6042.756871] LustreError: 310353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372596 with bad export cookie 16699451639384495226 [ 6042.763537] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.765039] LustreError: 310353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6043.016586] Lustre: server umount lustre-MDT0001 complete [ 6056.873870] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 6059.816657] Key type lgssc unregistered [ 6060.038567] LNet: 312938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6060.046939] LNetError: 312938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6060.059675] LNet: Removed LNI 192.168.201.104@tcp [ 6060.935939] Key type .llcrypt unregistered [ 6060.938591] Key type ._llcrypt unregistered [ 6073.803790] Key type ._llcrypt registered [ 6073.805841] Key type .llcrypt registered [ 6073.941649] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6075.434821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6075.449266] alg: No test for adler32 (adler32-zlib) [ 6076.499920] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 6076.737375] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 6078.464345] Key type lgssc registered [ 6079.808639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6129.088325] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6138.779159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6138.798183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.084911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6140.102735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6140.234603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6145.477341] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6154.586835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.772434] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6154.826303] Lustre: 315699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6155.266585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6159.746599] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6167.996596] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6174.382665] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6197.368613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6197.790476] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6198.030962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6204.782530] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6213.338970] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6220.260537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6220.274205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.292485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6220.773658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6220.777827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.824475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6221.982679] Lustre: server umount lustre-OST0000 complete [ 6226.913170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6226.918149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.941707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6231.008718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6231.011405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.922570] Lustre: server umount lustre-MDT0000 complete [ 6237.168551] LustreError: 315694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372791 with bad export cookie 8661264326868893249 [ 6237.195341] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6237.200111] LustreError: 315694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6237.651673] Lustre: server umount lustre-MDT0001 complete [ 6249.976861] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 6252.804664] Key type lgssc unregistered [ 6253.057474] LNet: 319272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6253.067512] LNetError: 319272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6254.123035] LNet: Removed LNI 192.168.201.104@tcp [ 6255.594045] Key type .llcrypt unregistered [ 6255.595556] Key type ._llcrypt unregistered [ 6282.760766] Key type ._llcrypt registered [ 6282.765869] Key type .llcrypt registered [ 6282.929493] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6283.908975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6283.948817] alg: No test for adler32 (adler32-zlib) [ 6285.546800] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 6285.896094] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 6287.575225] Key type lgssc registered [ 6288.990953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6297.643952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6297.675778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6299.272500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6304.298991] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6309.860810] LustreError: 320686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.891292] LustreError: 320686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6314.988332] LustreError: 320687:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.093929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6315.691214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6320.730204] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6328.980649] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6335.051850] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6343.079062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6343.425044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6348.651994] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6354.890573] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6371.310109] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.312565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6371.329680] Lustre: Skipped 2 previous similar messages [ 6371.351550] Lustre: Skipped 2 previous similar messages [ 6372.028851] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.4@tcp (stopping) [ 6375.865438] Lustre: server umount lustre-MDT0000 complete [ 6376.417471] LustreError: 323191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6376.428339] LustreError: 323191:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6380.851956] LustreError: 320666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372934 with bad export cookie 15914333250290714781 [ 6380.890124] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6381.537466] LustreError: 320681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.538567] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.560390] LustreError: 320681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6381.582944] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6385.568553] LustreError: 320682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6385.572358] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6385.592691] LustreError: 320682:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6385.618501] Lustre: Skipped 1 previous similar message [ 6390.691922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6390.703988] Lustre: Skipped 1 previous similar message [ 6395.359695] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6395.846269] Lustre: server umount lustre-MDT0001 complete [ 6403.473941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6414.443040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.581790] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdcdb11c5fcc426e9 [ 6425.602836] Lustre: MGC192.168.201.104@tcp: Connection restored to 0@lo (at 0@lo) [ 6425.907184] LustreError: 322531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.931866] LustreError: 322531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6426.021634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.108045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 6431.933920] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6443.034395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6443.667981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6445.673915] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6449.608430] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6457.897282] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6459.598802] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6465.944488] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6467.459943] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6472.176624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6472.197622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.216172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6472.227635] Lustre: Skipped 2 previous similar messages [ 6474.209101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6474.223694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.166679] Lustre: server umount lustre-OST0000 complete [ 6481.891653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6481.907454] Lustre: Skipped 1 previous similar message [ 6489.581811] Lustre: server umount lustre-MDT0000 complete [ 6492.142427] LustreError: 324317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6492.161570] LustreError: 324317:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6493.177954] LustreError: 320667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373047 with bad export cookie 15914333250290722537 [ 6493.192828] LustreError: 320667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6493.205666] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6493.631598] Lustre: server umount lustre-MDT0001 complete [ 6502.998940] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 6505.483107] Key type lgssc unregistered [ 6505.704762] LNet: 326993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6505.708537] LNetError: 326993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6505.727167] LNet: Removed LNI 192.168.201.104@tcp [ 6506.644592] Key type .llcrypt unregistered [ 6506.648300] Key type ._llcrypt unregistered [ 6528.133547] Key type ._llcrypt registered [ 6528.138867] Key type .llcrypt registered [ 6528.255683] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6529.109871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6529.195158] alg: No test for adler32 (adler32-zlib) [ 6530.384053] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 6530.636227] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 6532.311260] Key type lgssc registered [ 6533.360785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6540.192543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6540.218322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6541.569978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6546.093927] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6551.021841] LustreError: 328400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.037893] LustreError: 328400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6553.793351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.046660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6558.325138] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6564.952181] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6570.524525] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6574.567958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6574.576449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.609568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6576.608465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6576.610923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.601972] Lustre: server umount lustre-MDT0000 complete [ 6579.684037] LustreError: 328397:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.696779] LustreError: 328397:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6583.039983] LustreError: 328381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373137 with bad export cookie 16851723979355052776 [ 6583.048070] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6583.050102] LustreError: 328381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6583.535612] Lustre: server umount lustre-MDT0001 complete [ 6598.775252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6599.085861] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6599.136355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6599.408868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6604.524320] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6612.975990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.127590] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6613.198312] Lustre: 330988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6613.517839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6617.965914] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6624.247169] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6629.652961] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6638.826128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6638.964172] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6638.971992] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6638.978750] LustreError: 332871:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 6638.983702] LustreError: 332871:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6638.987308] LustreError: 332871:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6639.042746] Lustre: server umount lustre-OST0000 complete [ 6639.045082] LustreError: 332871:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6648.237760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6648.398964] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6648.566690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6653.952813] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6658.283693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 6660.945093] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6670.479094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6670.609041] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6670.728467] Lustre: lustre-OST0001: new disk, initializing [ 6670.734890] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6670.743763] Lustre: 334412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6670.859651] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6677.912545] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6680.099469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6680.106243] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6680.143362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6684.309865] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6690.279166] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6690.284900] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.300716] LustreError: Skipped 1 previous similar message [ 6690.323267] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6690.325569] Lustre: Skipped 1 previous similar message [ 6690.345436] Lustre: Skipped 2 previous similar messages [ 6692.955563] Lustre: server umount lustre-OST0001 complete [ 6695.399054] LustreError: 333338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.422057] LustreError: 333338:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6699.489527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6699.505275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6699.526319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6699.536130] Lustre: Skipped 1 previous similar message [ 6700.524536] LustreError: 333340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6700.550503] LustreError: 333340:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6702.706987] Lustre: server umount lustre-OST0000 complete [ 6710.751915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6710.759884] LustreError: Skipped 1 previous similar message [ 6710.766194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6710.766202] Lustre: Skipped 1 previous similar message [ 6710.780530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.802979] Lustre: Skipped 3 previous similar messages [ 6713.214587] Lustre: server umount lustre-MDT0000 complete [ 6715.879978] LustreError: 330999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.920662] LustreError: 330999:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6718.099585] LustreError: 331727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373272 with bad export cookie 16851723979355053329 [ 6718.101084] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6718.115648] LustreError: 331727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6718.683883] Lustre: server umount lustre-MDT0001 complete [ 6732.799780] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 6736.268287] Key type lgssc unregistered [ 6736.519292] LNet: 336371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6736.537574] LNetError: 336371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6736.561846] LNet: Removed LNI 192.168.201.104@tcp [ 6737.378221] Key type .llcrypt unregistered [ 6737.380398] Key type ._llcrypt unregistered [ 6819.163563] Key type ._llcrypt registered [ 6819.165637] Key type .llcrypt registered [ 6819.239856] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 6835.085038] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 6836.681212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6836.701431] alg: No test for adler32 (adler32-zlib) [ 6837.665804] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 6837.864941] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 6839.511291] Key type lgssc registered [ 6840.516213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6882.372407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6882.402210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.775641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6883.831449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6883.970578] Lustre: lustre-MDT0000: new disk, initializing [ 6884.153220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6884.173869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6889.458931] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6903.440953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6903.655955] Lustre: 341309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6903.739996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6903.746348] Lustre: Skipped 1 previous similar message [ 6903.827192] Lustre: lustre-MDT0001: new disk, initializing [ 6903.962599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6904.033709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6904.047110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6910.007219] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6920.134345] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6926.350974] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6933.074390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.363750] Lustre: lustre-OST0000: new disk, initializing [ 6933.366856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6933.372212] Lustre: 343278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6933.526241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6940.630297] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 6942.864797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6942.885183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6943.033904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6950.300487] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6956.323334] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6956.732347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6962.295248] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6962.598784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6968.804169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6968.806239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6968.828320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6968.839149] Lustre: Skipped 1 previous similar message [ 6971.282555] Lustre: server umount lustre-OST0000 complete [ 6975.458661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6975.473632] LustreError: Skipped 1 previous similar message [ 6975.478884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.491049] Lustre: Skipped 1 previous similar message [ 6975.503376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6979.041894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.044909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6979.086311] Lustre: Skipped 1 previous similar message [ 6981.744884] Lustre: server umount lustre-MDT0000 complete [ 6985.812299] LustreError: 342112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373539 with bad export cookie 9417995105013944766 [ 6985.814653] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6985.822201] LustreError: 342112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6986.284899] Lustre: server umount lustre-MDT0001 complete [ 6995.843912] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:12:27 (1789373547) [ 7003.355310] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:12:35 (1789373555) [ 7012.065156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.544474] LustreError: 346034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7012.566725] LustreError: 346034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7012.648020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7017.953967] LustreError: 346035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7019.107732] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7022.052488] LustreError: 346034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7027.171170] LustreError: 346035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.671498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.000194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7034.028811] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7041.647524] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7047.724356] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.587061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.028155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.431624] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7071.572209] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7079.466606] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7081.350486] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7082.885862] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7084.654931] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7086.419870] Lustre: Failing over lustre-MDT0000 [ 7086.701700] Lustre: server umount lustre-MDT0000 complete [ 7088.815959] LustreError: 346029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7089.132243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7089.147448] Lustre: Skipped 2 previous similar messages [ 7099.051524] LustreError: 346031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.078692] LustreError: 346031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7105.503393] Lustre: 339527:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789373643/real 1789373643] req@00000000e36ff521 x1876293992914816/t0(0) o400->MGC192.168.201.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789373659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7105.526936] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7106.066254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7115.416924] LustreError: 346030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.456615] LustreError: 346030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 7115.749064] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x82b370a63ab9594a [ 7115.775223] LustreError: 349240:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.104@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000d0a0bac2) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7115.804032] Lustre: MGC192.168.201.104@tcp: Connection restored to 0@lo (at 0@lo) [ 7116.094225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.147623] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7120.540106] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7121.289060] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7121.397381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7121.441024] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7121.491769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7131.250793] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7133.412145] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7137.302551] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7139.553263] Lustre: Failing over lustre-MDT0000 [ 7139.900621] Lustre: server umount lustre-MDT0000 complete [ 7141.856923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.882277] Lustre: Skipped 2 previous similar messages [ 7151.261173] LustreError: 347524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7151.285604] LustreError: 347524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7158.240121] Lustre: 339527:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789373695/real 1789373695] req@00000000a959153b x1876293992937472/t0(0) o400->MGC192.168.201.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789373711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7158.294802] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7162.057573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7169.017141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7169.085731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7171.746393] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7174.119171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7174.127562] Lustre: Skipped 2 previous similar messages [ 7174.184429] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7174.227973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7174.372626] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7184.375830] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7186.463374] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7194.595829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7194.596396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7194.601771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7194.606584] Lustre: Skipped 2 previous similar messages [ 7199.729374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7199.745720] Lustre: Skipped 2 previous similar messages [ 7200.103709] Lustre: server umount lustre-MDT0000 complete [ 7203.470141] LustreError: 346015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373757 with bad export cookie 9417995105013947489 [ 7203.489236] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7203.832346] Lustre: server umount lustre-MDT0001 complete [ 7217.628416] Lustre: server umount lustre-OST0000 complete [ 7236.695635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.289494] LustreError: 352380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7237.329258] LustreError: 352380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [ 7237.406646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7243.529339] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7252.696055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.965473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7257.871953] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7265.562500] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7272.465924] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7282.055636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7289.606392] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7297.980378] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7307.078415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7317.472166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7317.475131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7317.489816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7317.511758] Lustre: Skipped 1 previous similar message [ 7317.541220] Lustre: Skipped 1 previous similar message [ 7329.247613] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7329.592342] Lustre: server umount lustre-OST0000 complete [ 7343.106839] Lustre: server umount lustre-MDT0000 complete [ 7348.016693] LustreError: 352361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373902 with bad export cookie 9417995105013948231 [ 7348.021611] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7348.030930] LustreError: 352361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7348.754860] Lustre: server umount lustre-MDT0001 complete [ 7363.580541] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 7366.854270] Key type lgssc unregistered [ 7367.223260] LNet: 355943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7367.237199] LNetError: 355943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7367.264866] LNet: Removed LNI 192.168.201.104@tcp [ 7368.413627] Key type .llcrypt unregistered [ 7368.415609] Key type ._llcrypt unregistered [ 7396.495532] Key type ._llcrypt registered [ 7396.498717] Key type .llcrypt registered [ 7396.628316] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7397.532747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7397.602190] alg: No test for adler32 (adler32-zlib) [ 7398.873828] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 7399.286794] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 7400.991456] Key type lgssc registered [ 7402.192853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7410.627294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7410.643286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7412.116833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.157492] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7422.439453] LustreError: 357469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.455283] LustreError: 357469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7427.552703] LustreError: 357470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7428.155597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7428.950498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7436.665816] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7447.384428] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7455.344239] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7463.250550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7463.842082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7471.581204] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7479.783391] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7482.296188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7497.698761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.705279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7497.715949] Lustre: Skipped 2 previous similar messages [ 7497.735425] Lustre: Skipped 2 previous similar messages [ 7498.169556] Lustre: server umount lustre-MDT0000 complete [ 7501.962404] LustreError: 360073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374055 with bad export cookie 987103306564221783 [ 7501.965111] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.978415] LustreError: 360073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7502.262670] Lustre: server umount lustre-MDT0001 complete [ 7515.579063] Lustre: server umount lustre-OST0000 complete [ 7531.131270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7531.682220] LustreError: 361205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7531.743829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7537.130787] LustreError: 361206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.581720] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7541.219765] LustreError: 361205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.219472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7547.666542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7554.085409] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7566.491176] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7575.171455] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7583.190649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7583.609819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.674802] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7601.017129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7601.196671] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7614.451400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7614.466616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7614.483400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7616.484139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.484659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7616.497766] Lustre: Skipped 1 previous similar message [ 7616.513652] Lustre: Skipped 2 previous similar messages [ 7620.725706] Lustre: server umount lustre-MDT0000 complete [ 7621.602201] LustreError: 361206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7621.632638] LustreError: 361206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7625.448680] LustreError: 361185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374179 with bad export cookie 987103306564222896 [ 7625.451235] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7625.458398] LustreError: 361185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7625.839480] Lustre: server umount lustre-MDT0001 complete [ 7639.582604] Lustre: server umount lustre-OST0000 complete [ 7658.469148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.823854] LustreError: 365120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.884708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.045636] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7673.762604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7678.392417] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7687.173542] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7692.928322] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7700.481382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7700.890658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7700.900621] Lustre: Skipped 1 previous similar message [ 7704.018236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7708.499198] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7718.445472] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7724.513225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7724.515065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7724.524014] LustreError: Skipped 1 previous similar message [ 7724.528125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7724.543695] Lustre: Skipped 1 previous similar message [ 7724.565793] Lustre: Skipped 1 previous similar message [ 7725.681360] Lustre: server umount lustre-OST0000 complete [ 7734.753171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7734.780183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.789390] Lustre: Skipped 1 previous similar message [ 7737.711690] Lustre: server umount lustre-MDT0000 complete [ 7739.884091] LustreError: 366167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7739.923184] LustreError: 366167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7742.668451] LustreError: 365101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374296 with bad export cookie 987103306564223918 [ 7742.676780] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7742.707132] LustreError: 365101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7743.592435] Lustre: server umount lustre-MDT0001 complete [ 7760.166768] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 7766.138603] Key type lgssc unregistered [ 7766.362862] LNet: 368654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7766.367547] LNetError: 368654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7766.385532] LNet: Removed LNI 192.168.201.104@tcp [ 7767.035411] Key type .llcrypt unregistered [ 7767.037126] Key type ._llcrypt unregistered [ 7799.292343] Key type ._llcrypt registered [ 7799.295355] Key type .llcrypt registered [ 7799.469393] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7801.276872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7801.364099] alg: No test for adler32 (adler32-zlib) [ 7802.647532] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 7802.961321] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 7804.673057] Key type lgssc registered [ 7806.131524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7816.823372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7816.858690] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.867030] LustreError: 370094:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7816.893794] LustreError: 370094:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7816.903156] LustreError: 370094:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7816.918071] LustreError: 370094:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7816.933366] LustreError: 370094:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7825.956022] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7825.961950] LustreError: 370492:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7825.977598] LustreError: 370492:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7825.986296] LustreError: 370492:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7825.991508] LustreError: 370492:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7825.997845] LustreError: 370492:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7846.151438] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 7848.658843] Key type lgssc unregistered [ 7848.869745] LNet: 371495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7848.876281] LNetError: 371495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7848.895412] LNet: Removed LNI 192.168.201.104@tcp [ 7849.636808] Key type .llcrypt unregistered [ 7849.638891] Key type ._llcrypt unregistered [ 7871.888754] Key type ._llcrypt registered [ 7871.890141] Key type .llcrypt registered [ 7872.067710] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7890.581817] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7892.156381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7892.306248] alg: No test for adler32 (adler32-zlib) [ 7893.539552] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 7893.850455] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 7895.546303] Key type lgssc registered [ 7897.329119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7934.382454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7934.421708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.664750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7935.706481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7935.817158] Lustre: lustre-MDT0000: new disk, initializing [ 7935.912101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7935.928528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7941.811238] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7953.325045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7953.629219] Lustre: 374647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7953.836050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7953.851761] Lustre: Skipped 1 previous similar message [ 7954.015931] Lustre: lustre-MDT0001: new disk, initializing [ 7954.208874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7954.293591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7954.309497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7960.023947] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7970.546961] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7977.131456] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7984.010596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7984.465375] Lustre: lustre-OST0000: new disk, initializing [ 7984.470491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7984.482471] Lustre: 376618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7984.635543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.897417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7991.913678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7992.166530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7993.672477] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8003.613357] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8008.854084] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8009.243596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8016.347154] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8016.635849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8023.008696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8023.010367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8023.029734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8023.065391] Lustre: Skipped 1 previous similar message [ 8025.011435] Lustre: server umount lustre-OST0000 complete [ 8031.206702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8031.217960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.241183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.250557] Lustre: Skipped 1 previous similar message [ 8033.259963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.269404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8033.307880] Lustre: Skipped 1 previous similar message [ 8036.355854] Lustre: server umount lustre-MDT0000 complete [ 8040.944071] LustreError: 374642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374594 with bad export cookie 8115229908323135768 [ 8040.947862] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8040.948466] LustreError: 374642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8041.326360] Lustre: server umount lustre-MDT0001 complete [ 8055.197109] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8057.728885] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:30:09 (1789374609) [ 8064.786396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.536169] LustreError: 379081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8065.568139] LustreError: 379081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8065.683935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8071.142293] LustreError: 379082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8071.538840] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8075.245062] LustreError: 379081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8080.358015] LustreError: 379082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8082.011302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8083.105315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8088.626828] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8097.514736] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8103.779175] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8112.779059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8113.020539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8124.393662] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8133.516547] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8141.320386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8141.645296] Lustre: lustre-OST0001: new disk, initializing [ 8141.654681] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8141.662070] Lustre: 381919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8141.793478] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8147.688681] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8151.208926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8151.228542] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8151.337340] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8156.574693] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8161.761811] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8161.763613] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8161.775716] LustreError: Skipped 1 previous similar message [ 8161.788890] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8161.801243] Lustre: Skipped 1 previous similar message [ 8161.833827] Lustre: Skipped 1 previous similar message [ 8166.467763] Lustre: server umount lustre-OST0001 complete [ 8166.903618] LustreError: 381632:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8166.925545] LustreError: 381632:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8177.121666] LustreError: 380902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8177.149611] LustreError: 380902:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8186.848609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8186.862119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.876542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8186.878584] Lustre: Skipped 1 previous similar message [ 8190.242819] Lustre: server umount lustre-OST0000 complete [ 8195.552672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8195.564268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.574551] Lustre: Skipped 1 previous similar message [ 8195.580665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8195.583678] Lustre: Skipped 2 previous similar messages [ 8208.351214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8208.989754] Lustre: server umount lustre-MDT0000 complete [ 8212.973690] LustreError: 379077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8212.996432] LustreError: 379077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8213.093418] LustreError: 379062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374767 with bad export cookie 8115229908323136650 [ 8213.118518] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8219.579844] Lustre: server umount lustre-MDT0001 complete [ 8230.981772] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8234.599886] Key type lgssc unregistered [ 8234.889185] LNet: 383905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8234.899466] LNetError: 383905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8234.921965] LNet: Removed LNI 192.168.201.104@tcp [ 8235.736623] Key type .llcrypt unregistered [ 8235.739037] Key type ._llcrypt unregistered [ 8284.434300] Key type ._llcrypt registered [ 8284.437273] Key type .llcrypt registered [ 8284.553587] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8285.730151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8285.757121] alg: No test for adler32 (adler32-zlib) [ 8286.825983] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 8287.109900] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 8288.800712] Key type lgssc registered [ 8290.290477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8298.825325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8298.852957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.180605] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8300.208389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8300.453817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8306.664905] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8318.642780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8318.890732] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8318.997895] Lustre: 386702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8319.406881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8325.923855] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8334.488917] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8339.577841] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8342.499388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8342.501675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8342.522870] Lustre: Skipped 1 previous similar message [ 8347.616348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8347.625593] Lustre: Skipped 2 previous similar messages [ 8352.736752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8352.739619] Lustre: Skipped 1 previous similar message [ 8356.319169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8356.515907] Lustre: server umount lustre-MDT0000 complete [ 8359.850590] LustreError: 386695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374913 with bad export cookie 482236855065805252 [ 8359.859151] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8359.865581] LustreError: 386695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8360.217595] Lustre: server umount lustre-MDT0001 complete [ 8368.324759] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:35:20 (1789374920) [ 8374.812093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.285149] LustreError: 389215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.299974] LustreError: 389215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8375.353178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.393951] LustreError: 389216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.767641] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8385.508349] LustreError: 389215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8389.671157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.204338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8395.036426] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8403.340944] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8408.897788] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8415.677261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8415.801116] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8415.919486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8421.957947] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8429.242477] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8436.186189] Lustre: Permanently deactivating lustre-OST0000 [ 8436.213589] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8444.498819] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8446.216910] LustreError: 392201:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8446.224730] LustreError: 392201:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8446.228473] LustreError: 392201:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8449.370324] LustreError: 392250:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8449.380993] LustreError: 392250:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8457.989602] Lustre: server umount lustre-OST0000 complete [ 8460.690416] LustreError: 392499:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8460.700213] LustreError: 392499:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8465.376519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8465.384113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8465.394764] Lustre: Skipped 1 previous similar message [ 8465.405781] Lustre: Skipped 1 previous similar message [ 8470.497308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8470.503737] Lustre: Skipped 1 previous similar message [ 8477.151225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8477.491718] Lustre: server umount lustre-MDT0000 complete [ 8480.746072] LustreError: 389942:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8480.776596] LustreError: 389942:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8481.422903] LustreError: 389196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375035 with bad export cookie 482236855065805833 [ 8481.430817] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8481.434348] LustreError: 389196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8491.159632] Lustre: server umount lustre-MDT0001 complete [ 8501.165156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.141415] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8509.068223] LustreError: 393660:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8515.882175] Lustre: 393999:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.104@tcp [ 8518.062570] Lustre: server umount MGS complete [ 8531.103502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.628675] LustreError: 394724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.662648] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8531.667454] Lustre: Skipped 1 previous similar message [ 8531.721585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8536.369488] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8542.180187] LustreError: 394725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8542.204442] LustreError: 394725:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8547.433292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.729658] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8547.825500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8552.197928] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8560.949683] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8566.017089] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8567.449778] Lustre: Permanently reactivating lustre-OST0000 [ 8567.463626] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8567.467510] Lustre: Skipped 2 previous similar messages [ 8596.453286] LustreError: 396549:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8604.550287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8604.910425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.747144] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8620.779869] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8626.657512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8626.663402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8626.671571] LustreError: Skipped 1 previous similar message [ 8626.673095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8626.673102] Lustre: Skipped 2 previous similar messages [ 8626.720477] Lustre: Skipped 1 previous similar message [ 8628.351778] Lustre: server umount lustre-OST0000 complete [ 8634.848352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8634.862919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8638.508081] Lustre: server umount lustre-MDT0000 complete [ 8642.017825] LustreError: 394725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8642.040331] LustreError: 394725:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8642.310358] LustreError: 394705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375196 with bad export cookie 482236855065806883 [ 8642.331598] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8642.750392] Lustre: server umount lustre-MDT0001 complete [ 8654.535472] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8657.355983] Key type lgssc unregistered [ 8657.775911] LNet: 399134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8657.781696] LNetError: 399134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8657.803932] LNet: Removed LNI 192.168.201.104@tcp [ 8658.870828] Key type .llcrypt unregistered [ 8658.872709] Key type ._llcrypt unregistered [ 8678.278255] Key type ._llcrypt registered [ 8678.280462] Key type .llcrypt registered [ 8678.419894] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8695.723943] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8696.927046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8697.043776] alg: No test for adler32 (adler32-zlib) [ 8698.240563] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 8698.541283] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 8700.224084] Key type lgssc registered [ 8701.287915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8739.678785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8739.700044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8740.961816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8740.981175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8741.063933] Lustre: lustre-MDT0000: new disk, initializing [ 8741.154886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8741.173595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8745.801409] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8756.342553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8756.480907] Lustre: 402291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8756.559626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8756.564033] Lustre: Skipped 1 previous similar message [ 8756.606420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8756.695813] Lustre: lustre-MDT0001: new disk, initializing [ 8756.826093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8756.875757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8756.903685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8761.295220] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8772.333909] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8779.507629] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8786.179434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8786.590827] Lustre: lustre-OST0000: new disk, initializing [ 8786.599781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8786.611764] Lustre: 404259:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8786.776176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8793.470964] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8796.278838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8796.295782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8796.487948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8802.714880] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8808.299581] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8808.520206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8814.039491] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8814.345960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8817.123657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8817.131515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8817.139106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8822.391595] Lustre: server umount lustre-OST0000 complete [ 8827.361822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8827.366813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8827.381816] Lustre: Skipped 2 previous similar messages [ 8827.396981] Lustre: Skipped 1 previous similar message [ 8832.480571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8832.489567] Lustre: Skipped 2 previous similar messages [ 8837.603403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8837.608628] Lustre: Skipped 1 previous similar message [ 8841.183118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8841.681400] Lustre: server umount lustre-MDT0000 complete [ 8842.727697] LustreError: 403372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8842.743157] LustreError: 403372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8846.010485] LustreError: 402281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375400 with bad export cookie 15514332260187213945 [ 8846.012665] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8846.028841] LustreError: 402281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8846.774582] Lustre: server umount lustre-MDT0001 complete [ 8859.106611] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8861.311449] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:43:32 (1789375412) [ 8869.512622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8869.897498] LustreError: 406720:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8869.989104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8874.979606] LustreError: 406721:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8875.549565] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8879.076436] LustreError: 406720:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8884.198209] LustreError: 406721:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8884.637207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8885.106962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8889.811265] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8898.589466] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8906.429435] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8913.372485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8914.036321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8921.457639] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8929.204336] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8931.708699] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8935.343730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8941.032940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8941.040212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8941.052145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8941.063922] Lustre: Skipped 1 previous similar message [ 8946.932374] Lustre: server umount lustre-OST0000 complete [ 8951.265832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.283886] Lustre: Skipped 2 previous similar messages [ 8957.668583] Lustre: server umount lustre-MDT0000 complete [ 8961.099530] LustreError: 406702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375515 with bad export cookie 15514332260187214827 [ 8961.111207] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8961.113733] LustreError: 406702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8961.759982] Lustre: server umount lustre-MDT0001 complete [ 8976.510221] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8979.979455] Key type lgssc unregistered [ 8980.397652] LNet: 410449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8980.413945] LNetError: 410449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8980.466962] LNet: Removed LNI 192.168.201.104@tcp [ 8981.370316] Key type .llcrypt unregistered [ 8981.372420] Key type ._llcrypt unregistered [ 9006.874443] Key type ._llcrypt registered [ 9006.875995] Key type .llcrypt registered [ 9006.963718] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9007.271444] Key type .llcrypt unregistered [ 9007.275643] Key type ._llcrypt unregistered [ 9026.322871] Key type ._llcrypt registered [ 9026.325895] Key type .llcrypt registered [ 9026.416163] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9027.639637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9027.762912] alg: No test for adler32 (adler32-zlib) [ 9028.838499] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 9029.150739] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9030.903504] Key type lgssc registered [ 9032.379275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9042.760456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9042.789879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9044.498708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9049.854966] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9054.695582] LustreError: 412655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9054.716559] LustreError: 412655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9058.276125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9058.507528] LustreError: 412656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9058.528955] LustreError: 412656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9058.547775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9063.908760] LustreError: 412655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9063.940309] LustreError: 412655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9065.711289] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9069.037468] LustreError: 412656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9069.056930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9069.069363] LustreError: 412656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9074.065787] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9074.144715] LustreError: 412655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9074.160678] LustreError: 412655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9081.787721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9082.403504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9087.323465] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9097.495229] Lustre: server umount lustre-OST0000 complete [ 9102.816482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9102.830141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.854401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9102.869192] Lustre: Skipped 1 previous similar message [ 9107.940730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9110.119542] Lustre: server umount lustre-MDT0000 complete [ 9113.059993] LustreError: 412655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9113.085893] LustreError: 412655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9113.704125] Lustre: 411966:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789375651/real 1789375651] req@000000009c59442d x1876296291345920/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789375667 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9113.722827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.730989] Lustre: Skipped 1 previous similar message [ 9114.353970] LustreError: 412637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375668 with bad export cookie 9650813166669959619 [ 9114.362318] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9114.379759] LustreError: 412637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9114.720830] Lustre: server umount lustre-MDT0001 complete [ 9126.908659] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9129.988982] Key type lgssc unregistered [ 9130.306203] LNet: 415803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9130.325655] LNetError: 415803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9130.354915] LNet: Removed LNI 192.168.201.104@tcp [ 9131.340229] Key type .llcrypt unregistered [ 9131.345573] Key type ._llcrypt unregistered [ 9156.853587] Key type ._llcrypt registered [ 9156.856182] Key type .llcrypt registered [ 9156.967574] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9158.199914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9158.417391] alg: No test for adler32 (adler32-zlib) [ 9159.600837] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 9159.863756] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9161.575276] Key type lgssc registered [ 9162.819556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9171.196739] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9171.210800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9186.783465] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9191.904519] LustreError: 417189:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.104@tcp: failed processing log, type 4: rc = -110 [ 9218.593119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9224.510198] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9231.047737] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9237.070560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.592901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9237.688108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9242.006468] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9242.603858] LustreError: 417214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9242.620313] LustreError: 417214:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9247.714978] LustreError: 417215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9250.379850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9250.714128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9255.234340] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9261.713656] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9266.594917] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9273.317799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9273.322129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9273.337748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9274.340349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9274.342487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9274.359787] Lustre: Skipped 1 previous similar message [ 9278.877587] Lustre: server umount lustre-OST0000 complete [ 9284.592211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9284.607584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9284.625282] Lustre: Skipped 1 previous similar message [ 9284.646807] Lustre: Skipped 1 previous similar message [ 9288.669289] Lustre: server umount lustre-MDT0000 complete [ 9289.701244] LustreError: 418233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9289.729355] LustreError: 418233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9292.127253] LustreError: 417197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375846 with bad export cookie 1786068898127473659 [ 9292.134816] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9292.136092] LustreError: 417197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9292.437481] Lustre: server umount lustre-MDT0001 complete [ 9301.996888] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9304.305843] Key type lgssc unregistered [ 9304.557723] LNet: 420758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9304.572117] LNetError: 420758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9304.606041] LNet: Removed LNI 192.168.201.104@tcp [ 9305.576916] Key type .llcrypt unregistered [ 9305.579092] Key type ._llcrypt unregistered [ 9325.973802] Key type ._llcrypt registered [ 9325.975753] Key type .llcrypt registered [ 9326.059772] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9327.069063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9327.155779] alg: No test for adler32 (adler32-zlib) [ 9328.205521] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 9328.414745] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9330.023710] Key type lgssc registered [ 9331.101855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9338.383408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9338.410541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9339.945528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9344.317430] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9349.092064] LustreError: 422153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9349.112580] LustreError: 422153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9352.725360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9352.980306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9357.396893] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9365.337215] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9371.142340] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9379.280728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9379.747377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.932450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9387.681274] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9397.637696] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9402.351261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9402.353169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9402.381900] Lustre: Skipped 2 previous similar messages [ 9402.401479] Lustre: Skipped 2 previous similar messages [ 9404.625275] LustreError: 423184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9404.751120] Lustre: server umount lustre-MDT0000 complete [ 9407.460457] LustreError: 422154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9407.491829] LustreError: 422154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9407.696987] LustreError: 422133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.4@tcp arrived at 1789375961 with bad export cookie 16653970769418482358 [ 9409.825560] LustreError: 422135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375963 with bad export cookie 16653970769418481462 [ 9409.846810] LustreError: 422135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9409.856207] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9412.580885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9412.587828] LustreError: 422881:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9412.599327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9412.607878] LustreError: 422881:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9414.642620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9416.256406] Lustre: server umount lustre-OST0000 complete [ 9419.756343] LustreError: 422148:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9419.789455] LustreError: 422148:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9424.366995] Lustre: server umount lustre-MDT0001 complete [ 9436.023586] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9438.871828] Key type lgssc unregistered [ 9439.120672] LNet: 425836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9439.136601] LNetError: 425836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9439.154864] LNet: Removed LNI 192.168.201.104@tcp [ 9439.973770] Key type .llcrypt unregistered [ 9439.975484] Key type ._llcrypt unregistered [ 9464.567221] Key type ._llcrypt registered [ 9464.569157] Key type .llcrypt registered [ 9464.665290] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9465.746637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9465.857195] alg: No test for adler32 (adler32-zlib) [ 9466.878215] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 9467.150878] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9468.855627] Key type lgssc registered [ 9469.989253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9478.675278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9478.698692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9480.144453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9484.804867] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9490.403205] LustreError: 427248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9490.435066] LustreError: 427248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9494.242090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9494.563675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9499.556349] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9507.357578] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9512.628730] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9518.255838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9518.624365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9525.749922] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9525.894018] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9534.075971] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9540.577262] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9540.587585] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9540.609046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9541.602036] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9541.606340] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9541.625699] Lustre: Skipped 2 previous similar messages [ 9544.898210] Lustre: server umount lustre-MDT0001 complete [ 9545.883580] LustreError: 427245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.854383] LustreError: 427248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.902486] LustreError: 427248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9556.965690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9556.966586] LustreError: 427249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9556.974276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9556.975122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9558.800948] Lustre: server umount lustre-OST0000 complete [ 9562.083658] LustreError: 427248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9564.499571] Lustre: server umount lustre-MDT0000 complete [ 9580.693372] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9585.192034] Key type lgssc unregistered [ 9585.614734] LNet: 430937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9585.618902] LNetError: 430937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9585.645413] LNet: Removed LNI 192.168.201.104@tcp [ 9587.127137] Key type .llcrypt unregistered [ 9587.128679] Key type ._llcrypt unregistered [ 9615.400588] Key type ._llcrypt registered [ 9615.407432] Key type .llcrypt registered [ 9615.523362] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9635.663303] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9636.988966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9637.175576] alg: No test for adler32 (adler32-zlib) [ 9638.569443] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 9638.979879] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9640.770356] Key type lgssc registered [ 9641.754653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9679.967386] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9698.617493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9698.656883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9700.201052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9700.239939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9700.387950] Lustre: lustre-MDT0000: new disk, initializing [ 9700.584179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9700.637484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9706.914756] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9726.397841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9726.601284] Lustre: lustre-OST0000: new disk, initializing [ 9726.607321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9726.611513] Lustre: Skipped 1 previous similar message [ 9726.615884] Lustre: 435972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9726.679651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9733.734390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9733.751584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9733.782965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9734.626928] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9748.082349] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9762.819895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9762.973938] Lustre: 434952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9763.042770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9763.149324] Lustre: lustre-MDT0001: new disk, initializing [ 9763.289208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9763.365505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9763.380351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9768.760454] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9778.999503] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9779.247414] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9784.390248] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9784.646387] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9790.153863] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.4@tcp (stopping) [ 9794.030717] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9794.037356] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9794.065542] Lustre: Skipped 1 previous similar message [ 9794.068144] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9795.229345] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.4@tcp (stopping) [ 9795.245738] Lustre: Skipped 1 previous similar message [ 9799.137928] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9799.143893] Lustre: Skipped 1 previous similar message [ 9804.255120] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9804.261852] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9804.269381] Lustre: Skipped 2 previous similar messages [ 9804.477341] Lustre: server umount lustre-MDT0001 complete [ 9814.513280] LustreError: 434961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9814.533870] LustreError: 434961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9819.620224] LustreError: 434959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9819.671976] LustreError: 434959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9824.736968] LustreError: 434965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9824.740569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.823598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9829.082702] Lustre: server umount lustre-OST0000 complete [ 9829.856107] LustreError: 434964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9840.156845] Lustre: server umount lustre-MDT0000 complete [ 9858.667310] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9863.428754] Key type lgssc unregistered [ 9863.878922] LNet: 439722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9863.892243] LNetError: 439722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9863.903371] LNet: Removed LNI 192.168.201.104@tcp [ 9865.290688] Key type .llcrypt unregistered [ 9865.297849] Key type ._llcrypt unregistered [ 9918.122680] Key type ._llcrypt registered [ 9918.124983] Key type .llcrypt registered [ 9918.240781] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9918.741304] Key type .llcrypt unregistered [ 9918.745498] Key type ._llcrypt unregistered [ 9938.662453] Key type ._llcrypt registered [ 9938.665169] Key type .llcrypt registered [ 9938.790273] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9939.907662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9940.023602] alg: No test for adler32 (adler32-zlib) [ 9941.509838] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [ 9941.853227] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9943.507813] Key type lgssc registered [ 9944.465736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9976.548762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9976.591580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9978.278609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9978.406167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9978.594530] Lustre: lustre-MDT0000: new disk, initializing [ 9978.970993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9979.010045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9987.444767] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10002.132548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10002.246717] Lustre: 443105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10002.297688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10002.301792] Lustre: Skipped 1 previous similar message [10002.391767] Lustre: lustre-MDT0001: new disk, initializing [10002.531325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10002.599766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10002.621186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10007.697953] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10018.324630] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10025.625913] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10033.153127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10033.492376] Lustre: lustre-OST0000: new disk, initializing [10033.498302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10033.513528] Lustre: 445076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10033.651991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10040.409494] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10041.952943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10041.963718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10042.104186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10050.270251] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10058.727850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10058.737639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10058.758944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10062.818274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10062.831254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10062.846776] Lustre: Skipped 1 previous similar message [10062.885835] Lustre: Skipped 1 previous similar message [10064.678297] Lustre: server umount lustre-MDT0000 complete [10069.644839] LustreError: 443098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789376623 with bad export cookie 16353297842639763676 [10069.646080] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10069.665659] LustreError: 443098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10070.158632] Lustre: server umount lustre-MDT0001 complete [10084.492200] Lustre: server umount lustre-OST0000 complete [10097.919849] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:04:09 (1789376649) [10107.727845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.618788] LustreError: 447105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.641341] LustreError: 447105:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10108.775057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10114.018398] LustreError: 447106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.502545] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10118.128573] LustreError: 447105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.233671] LustreError: 447106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10124.009764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10124.289965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10129.350269] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10138.095324] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10144.858930] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10166.385041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10166.613504] Lustre: Found index 0 for lustre-OST0000, updating log [10166.789204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10175.595077] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10179.148163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10184.068304] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10194.401511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10194.403838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10194.410358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10194.410367] Lustre: Skipped 1 previous similar message [10194.445712] Lustre: Skipped 1 previous similar message [10197.487539] Lustre: server umount lustre-OST0000 complete [10216.417731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10216.434286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10216.452528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10216.457154] Lustre: Skipped 3 previous similar messages [10221.040232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10221.054102] Lustre: Skipped 2 previous similar messages [10221.555980] Lustre: server umount lustre-MDT0000 complete [10222.052945] LustreError: 447834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10222.071696] LustreError: 447834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10227.348471] LustreError: 449700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789376781 with bad export cookie 16353297842639765069 [10227.353216] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10227.358053] LustreError: 449700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10227.909982] Lustre: server umount lustre-MDT0001 complete [10230.751607] LNet: 1 local NIs in recovery (showing 1): 192.168.201.104@tcp [10238.142496] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:06:29 (1789376789) [10877.423899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.892859] LustreError: 453520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10877.903562] LustreError: 453520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10878.003081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10882.498211] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10883.041837] LustreError: 453521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10887.149067] LustreError: 453520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.472233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10891.861828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10897.164522] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10906.641746] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10912.383831] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10932.525926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10932.873034] Lustre: Found index 0 for lustre-OST0000, updating log [10933.309109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10933.369818] LNet: 1 local NIs in recovery (showing 1): 192.168.201.104@tcp [10941.241313] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10951.050511] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10953.442505] Lustre: server umount lustre-OST0000 complete [10958.054596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.067277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10958.072715] Lustre: Skipped 2 previous similar messages [10958.098366] Lustre: Skipped 1 previous similar message [10959.138304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10959.149656] Lustre: Skipped 1 previous similar message [10961.189495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10961.203945] Lustre: Skipped 3 previous similar messages [10963.508361] Lustre: server umount lustre-MDT0000 complete [10964.259563] LustreError: 453520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10964.284142] LustreError: 453520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10968.352498] LustreError: 453502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377522 with bad export cookie 16353297842639766154 [10968.367954] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10968.383799] LustreError: 453502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10968.927604] Lustre: server umount lustre-MDT0001 complete [10993.952326] LNet: 1 local NIs in recovery (showing 1): 192.168.201.104@tcp [11003.151628] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [11021.608928] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11062.083980] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:20:13 (1789377613) [11083.232702] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [11087.492960] Key type lgssc unregistered [11087.988623] LNet: 460156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11087.998349] LNetError: 460156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11195.558766] LNet: Removed LNI 192.168.201.104@tcp [11196.807946] Key type .llcrypt unregistered [11196.811573] Key type ._llcrypt unregistered [11218.674467] Key type ._llcrypt registered [11218.676894] Key type .llcrypt registered [11218.886996] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11220.472729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11220.590065] alg: No test for adler32 (adler32-zlib) [11222.056516] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [11222.272919] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [11223.927507] Key type lgssc registered [11225.714162] Lustre: Echo OBD driver; http://www.lustre.org/ [11230.142116] LNet: Added LNI 192.168.201.104@tcp100 [8/256/0/180] [11230.785830] LNet: Added LNI 192.168.201.104@tcp101 [8/256/0/180] [11231.460465] LNet: Added LNI 192.168.201.104@tcp102 [8/256/0/180] [11232.558082] LNet: Added LNI 192.168.201.104@tcp103 [8/256/0/180] [11233.539249] LNet: Added LNI 192.168.201.104@tcp104 [8/256/0/180] [11234.814527] LNet: Added LNI 192.168.201.104@tcp105 [8/256/0/180] [11236.280892] LNet: Added LNI 192.168.201.104@tcp106 [8/256/0/180] [11237.076529] LNet: Added LNI 192.168.201.104@tcp107 [8/256/0/180] [11238.196979] LNet: Added LNI 192.168.201.104@tcp108 [8/256/0/180] [11239.100993] LNet: Added LNI 192.168.201.104@tcp109 [8/256/0/180] [11239.879335] LNet: Added LNI 192.168.201.104@tcp110 [8/256/0/180] [11240.939882] LNet: Added LNI 192.168.201.104@tcp111 [8/256/0/180] [11242.088683] LNet: Added LNI 192.168.201.104@tcp112 [8/256/0/180] [11243.120543] LNet: Added LNI 192.168.201.104@tcp113 [8/256/0/180] [11244.153691] LNet: Added LNI 192.168.201.104@tcp114 [8/256/0/180] [11245.361792] LNet: Added LNI 192.168.201.104@tcp115 [8/256/0/180] [11246.477952] LNet: Added LNI 192.168.201.104@tcp116 [8/256/0/180] [11247.435701] LNet: Added LNI 192.168.201.104@tcp117 [8/256/0/180] [11248.643650] LNet: Added LNI 192.168.201.104@tcp118 [8/256/0/180] [11250.057185] LNet: Added LNI 192.168.201.104@tcp119 [8/256/0/180] [11251.230026] LNet: Added LNI 192.168.201.104@tcp120 [8/256/0/180] [11252.699327] LNet: Added LNI 192.168.201.104@tcp121 [8/256/0/180] [11253.844955] LNet: Added LNI 192.168.201.104@tcp122 [8/256/0/180] [11255.346305] LNet: Added LNI 192.168.201.104@tcp123 [8/256/0/180] [11256.803759] LNet: Added LNI 192.168.201.104@tcp124 [8/256/0/180] [11258.308063] LNet: Added LNI 192.168.201.104@tcp125 [8/256/0/180] [11260.663861] LNet: Added LNI 192.168.201.104@tcp126 [8/256/0/180] [11261.681899] LNet: Added LNI 192.168.201.104@tcp127 [8/256/0/180] [11262.714335] LNet: Added LNI 192.168.201.104@tcp128 [8/256/0/180] [11263.419809] LNet: Added LNI 192.168.201.104@tcp129 [8/256/0/180] [11264.322835] LNet: Added LNI 192.168.201.104@tcp130 [8/256/0/180] [11265.476399] LNet: Added LNI 192.168.201.104@tcp131 [8/256/0/180] [11266.574599] LNet: Added LNI 192.168.201.104@tcp132 [8/256/0/180] [11267.948497] LNet: Added LNI 192.168.201.104@tcp133 [8/256/0/180] [11269.111810] LNet: Added LNI 192.168.201.104@tcp134 [8/256/0/180] [11270.065777] LNet: Added LNI 192.168.201.104@tcp135 [8/256/0/180] [11275.852828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11275.891821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11277.023269] Lustre: 463185:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.104@tcp: can't add NID 192.168.201.104@tcp130: rc = -75 [11277.132727] LNetError: 460643:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.104 [11277.145031] LNetError: Protocol error connecting to 192.168.201.104@tcp113 at host 192.168.201.104:988: is it running a compatible version of Lustre? [11277.158835] LNetError: Skipped 4 previous similar messages [11277.249609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11277.270487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11277.372815] Lustre: lustre-MDT0000: new disk, initializing [11277.499842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11277.515807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11277.683366] LNet: 9 local NIs in recovery (showing 9): 192.168.201.104@tcp121, 192.168.201.104@tcp122, 192.168.201.104@tcp123, 192.168.201.104@tcp124, 192.168.201.104@tcp125, 192.168.201.104@tcp126, 192.168.201.104@tcp127, 192.168.201.104@tcp128, 192.168.201.104@tcp129 [11278.756468] LNetError: 460645:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.104 [11278.776689] LNetError: 460645:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [11278.787542] LNetError: Protocol error connecting to 192.168.201.104@tcp121 at host 192.168.201.104:988: is it running a compatible version of Lustre? [11278.811494] LNetError: Skipped 8 previous similar messages [11280.807366] LNetError: 460644:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.104 [11280.816552] LNetError: 460644:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [11280.832125] LNetError: Protocol error connecting to 192.168.201.104@tcp121 at host 192.168.201.104:988: is it running a compatible version of Lustre? [11280.839275] LNetError: Skipped 4 previous similar messages [11282.017372] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11284.896897] LNetError: 463207:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.104 [11284.911183] LNetError: Protocol error connecting to 192.168.201.104@tcp121 at host 192.168.201.104:988: is it running a compatible version of Lustre? [11292.064578] LNetError: 463209:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.104 [11292.076717] LNetError: Protocol error connecting to 192.168.201.104@tcp121 at host 192.168.201.104:988: is it running a compatible version of Lustre? [11292.269404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11292.464790] Lustre: 463202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11292.565712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11292.571612] Lustre: Skipped 1 previous similar message [11292.691937] Lustre: lustre-MDT0001: new disk, initializing [11292.747249] Lustre: 464003:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.104@tcp: can't add NID 192.168.201.104@tcp130: rc = -75 [11292.755298] Lustre: 464003:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11292.906211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11292.979848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11292.990323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11297.700880] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11301.442488] Lustre: 464387:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.104@tcp: can't add NID 192.168.201.104@tcp130: rc = -75 [11301.456137] Lustre: 464387:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11308.386251] LNetError: 463204:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.104 [11308.405856] LNetError: Protocol error connecting to 192.168.201.104@tcp122 at host 192.168.201.104:988: is it running a compatible version of Lustre? [11308.777572] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11315.407551] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11322.633082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11322.875681] Lustre: lustre-OST0000: new disk, initializing [11322.885882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11322.891575] Lustre: 465182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11322.951987] Lustre: 465187:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.104@tcp: can't add NID 192.168.201.104@tcp130: rc = -75 [11322.956759] Lustre: 465187:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11322.990492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11324.844521] LNetError: 460643:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.104 [11324.851590] LNetError: 460643:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [11324.878393] LNetError: Protocol error connecting to 192.168.201.104@tcp120 at host 192.168.201.104:988: is it running a compatible version of Lustre? [11324.898369] LNetError: Skipped 11 previous similar messages [11329.963745] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11332.135269] Lustre: 465638:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.104@tcp: can't add NID 192.168.201.104@tcp130: rc = -75 [11332.151544] Lustre: 465638:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11332.241622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11332.248142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11332.396971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11337.119154] LNet: 1 local NIs in recovery (showing 1): 192.168.201.104@tcp121 [11338.913935] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11347.936896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11347.949416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11347.956742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11347.964281] Lustre: Skipped 1 previous similar message [11347.976456] Lustre: Skipped 1 previous similar message [11352.533933] Lustre: server umount lustre-OST0000 complete [11357.601586] LNetError: 463206:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.104 [11357.608426] LNetError: 463206:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [11357.612688] LNetError: Protocol error connecting to 192.168.201.104@tcp119 at host 192.168.201.104:988: is it running a compatible version of Lustre? [11357.620783] LNetError: Skipped 20 previous similar messages [11358.180330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11358.200730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11358.204717] Lustre: Skipped 1 previous similar message [11363.406874] Lustre: server umount lustre-MDT0000 complete [11368.418859] LustreError: 463217:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11368.443145] LustreError: 463217:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11368.880483] LustreError: 463193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377922 with bad export cookie 8303435807630736932 [11368.888344] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11368.910458] LustreError: 463193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11369.537805] Lustre: server umount lustre-MDT0001 complete [11382.404184] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [11385.513809] Key type lgssc unregistered [11386.062811] LNet: 467094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11386.079681] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11386.085888] LNet: Removed LNI 192.168.201.104@tcp [11387.103175] LNet: Removed LNI 192.168.201.104@tcp100 [11387.108743] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11387.115400] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11387.120357] LNet: Removed LNI 192.168.201.104@tcp101 [11388.127143] LNet: Removed LNI 192.168.201.104@tcp102 [11388.131960] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11388.138122] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11389.151111] LNet: Removed LNI 192.168.201.104@tcp103 [11390.175159] LNet: Removed LNI 192.168.201.104@tcp104 [11390.180569] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11390.188492] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11390.195321] LNet: Removed LNI 192.168.201.104@tcp105 [11391.263680] LNet: Removed LNI 192.168.201.104@tcp106 [11392.287548] LNet: Removed LNI 192.168.201.104@tcp107 [11393.311450] LNet: Removed LNI 192.168.201.104@tcp108 [11394.335811] LNet: Removed LNI 192.168.201.104@tcp109 [11394.344360] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11394.372345] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11394.387116] LNet: Removed LNI 192.168.201.104@tcp110 [11395.423784] LNet: Removed LNI 192.168.201.104@tcp111 [11396.447309] LNet: Removed LNI 192.168.201.104@tcp112 [11397.471399] LNet: Removed LNI 192.168.201.104@tcp113 [11398.495177] LNet: Removed LNI 192.168.201.104@tcp114 [11399.519255] LNet: Removed LNI 192.168.201.104@tcp115 [11400.543168] LNet: Removed LNI 192.168.201.104@tcp116 [11401.572948] LNet: Removed LNI 192.168.201.104@tcp117 [11402.591219] LNet: Removed LNI 192.168.201.104@tcp118 [11402.599404] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11402.614162] LNetError: 467094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11402.627249] LNet: Removed LNI 192.168.201.104@tcp119 [11403.679239] LNet: Removed LNI 192.168.201.104@tcp120 [11404.703184] LNet: Removed LNI 192.168.201.104@tcp121 [11405.727225] LNet: Removed LNI 192.168.201.104@tcp122 [11406.817375] LNet: Removed LNI 192.168.201.104@tcp123 [11407.839230] LNet: Removed LNI 192.168.201.104@tcp124 [11408.863267] LNet: Removed LNI 192.168.201.104@tcp125 [11409.887321] LNet: Removed LNI 192.168.201.104@tcp126 [11410.912055] LNet: Removed LNI 192.168.201.104@tcp127 [11411.935184] LNet: Removed LNI 192.168.201.104@tcp128 [11412.960157] LNet: Removed LNI 192.168.201.104@tcp129 [11412.962473] LNet: Removed LNI 192.168.201.104@tcp130 [11412.965631] LNet: Removed LNI 192.168.201.104@tcp131 [11412.972474] LNet: Removed LNI 192.168.201.104@tcp132 [11412.975457] LNet: Removed LNI 192.168.201.104@tcp133 [11412.982777] LNet: Removed LNI 192.168.201.104@tcp134 [11413.010577] LNet: Removed LNI 192.168.201.104@tcp135 [11413.907462] Key type .llcrypt unregistered [11413.913945] Key type ._llcrypt unregistered [11441.474827] Key type ._llcrypt registered [11441.477425] Key type .llcrypt registered [11441.642701] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [11461.625628] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11464.085302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11464.109299] alg: No test for adler32 (adler32-zlib) [11465.190821] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [11465.427393] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [11467.064485] Key type lgssc registered [11468.524759] Lustre: Echo OBD driver; http://www.lustre.org/ [11507.988551] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:27:38 (1789378058) [11532.788329] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [11535.491865] Key type lgssc unregistered [11535.907330] LNet: 471181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11535.914834] LNetError: 471181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11535.940562] LNet: Removed LNI 192.168.201.104@tcp [11537.081722] Key type .llcrypt unregistered [11537.084685] Key type ._llcrypt unregistered [11553.584118] Key type ._llcrypt registered [11553.585825] Key type .llcrypt registered [11553.730387] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11555.580255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11555.648395] alg: No test for adler32 (adler32-zlib) [11556.819065] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [11557.072893] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [11558.807285] Key type lgssc registered [11560.358437] Lustre: Echo OBD driver; http://www.lustre.org/ [11572.926493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11572.982286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11574.303252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11574.365554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11574.467609] Lustre: lustre-MDT0000: new disk, initializing [11574.549800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11574.565908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11580.381804] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11591.737705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11591.847219] Lustre: 472394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11591.886085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11591.888714] Lustre: Skipped 1 previous similar message [11591.941625] Lustre: lustre-MDT0001: new disk, initializing [11592.036104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11592.059116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11592.072594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11596.877933] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11605.787546] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11611.749734] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11617.816437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11618.280946] Lustre: lustre-OST0000: new disk, initializing [11618.287661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11618.294377] Lustre: 474362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11618.446320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11624.519949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11624.537480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11624.695805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11626.000575] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11637.764593] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11643.981147] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing lnet_if_list [11651.283787] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11651.340343] LNet: Added LNI 192.168.201.104@tcp10 [8/256/0/180] [11657.482123] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11657.521307] LNet: Added LNI 192.168.201.104@tcp11 [8/256/0/180] [11663.533125] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11663.584217] LNet: Added LNI 192.168.201.104@tcp12 [8/256/0/180] [11669.993283] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11670.040168] LNet: Added LNI 192.168.201.104@tcp13 [8/256/0/180] [11675.582191] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11675.640828] LNet: Added LNI 192.168.201.104@tcp14 [8/256/0/180] [11682.632271] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11682.697861] LNet: Added LNI 192.168.201.104@tcp15 [8/256/0/180] [11690.248631] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11690.318690] LNet: Added LNI 192.168.201.104@tcp16 [8/256/0/180] [11696.938746] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11696.970806] LNet: Added LNI 192.168.201.104@tcp17 [8/256/0/180] [11704.595589] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11704.666609] LNet: Added LNI 192.168.201.104@tcp18 [8/256/0/180] [11712.872943] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11712.894709] LNet: Added LNI 192.168.201.104@tcp19 [8/256/0/180] [11719.095712] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11719.117677] LNet: Added LNI 192.168.201.104@tcp20 [8/256/0/180] [11724.938524] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11724.974914] LNet: Added LNI 192.168.201.104@tcp21 [8/256/0/180] [11731.807782] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11731.868056] LNet: Added LNI 192.168.201.104@tcp22 [8/256/0/180] [11738.858242] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11738.919809] LNet: Added LNI 192.168.201.104@tcp23 [8/256/0/180] [11746.137730] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11746.180466] LNet: Added LNI 192.168.201.104@tcp24 [8/256/0/180] [11753.718839] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11753.775337] LNet: Added LNI 192.168.201.104@tcp25 [8/256/0/180] [11760.628094] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11760.668029] LNet: Added LNI 192.168.201.104@tcp26 [8/256/0/180] [11767.711575] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11767.739753] LNet: Added LNI 192.168.201.104@tcp27 [8/256/0/180] [11774.506939] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11774.532532] LNet: Added LNI 192.168.201.104@tcp28 [8/256/0/180] [11782.232176] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11782.296594] LNet: Added LNI 192.168.201.104@tcp29 [8/256/0/180] [11791.200627] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11791.247957] LNet: Added LNI 192.168.201.104@tcp30 [8/256/0/180] [11799.855600] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11799.897867] LNet: Added LNI 192.168.201.104@tcp31 [8/256/0/180] [11806.597356] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11806.631977] LNet: Added LNI 192.168.201.104@tcp32 [8/256/0/180] [11813.286363] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11813.318236] LNet: Added LNI 192.168.201.104@tcp33 [8/256/0/180] [11820.953622] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11821.006160] LNet: Added LNI 192.168.201.104@tcp34 [8/256/0/180] [11828.287063] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11828.325120] LNet: Added LNI 192.168.201.104@tcp35 [8/256/0/180] [11834.935464] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11834.956673] LNet: Added LNI 192.168.201.104@tcp36 [8/256/0/180] [11843.491621] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11843.547766] LNet: Added LNI 192.168.201.104@tcp37 [8/256/0/180] [11854.576332] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11854.628462] LNet: Added LNI 192.168.201.104@tcp38 [8/256/0/180] [11864.165869] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11864.216328] LNet: Added LNI 192.168.201.104@tcp39 [8/256/0/180] [11874.019314] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11874.045227] LNet: Added LNI 192.168.201.104@tcp40 [8/256/0/180] [11885.644383] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11885.685249] LNet: Added LNI 192.168.201.104@tcp41 [8/256/0/180] [11895.054769] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11895.091286] LNet: Added LNI 192.168.201.104@tcp42 [8/256/0/180] [11903.977702] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11904.005682] LNet: Added LNI 192.168.201.104@tcp43 [8/256/0/180] [11913.809838] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11913.890587] LNet: Added LNI 192.168.201.104@tcp44 [8/256/0/180] [11925.299435] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11925.374327] LNet: Added LNI 192.168.201.104@tcp45 [8/256/0/180] [11935.382095] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11935.397810] LNet: Added LNI 192.168.201.104@tcp46 [8/256/0/180] [11946.400795] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11946.428415] LNet: Added LNI 192.168.201.104@tcp47 [8/256/0/180] [11957.387186] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11957.406821] LNet: Added LNI 192.168.201.104@tcp48 [8/256/0/180] [11967.722196] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11967.779621] LNet: Added LNI 192.168.201.104@tcp49 [8/256/0/180] [11981.011726] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11981.058989] LNet: Added LNI 192.168.201.104@tcp50 [8/256/0/180] [12012.002371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12012.009315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.028518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12014.061337] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12014.066413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12014.082033] Lustre: Skipped 1 previous similar message [12014.093476] Lustre: Skipped 2 previous similar messages [12019.170375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12019.172969] Lustre: Skipped 2 previous similar messages [12024.288484] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12024.290757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12024.323707] Lustre: Skipped 2 previous similar messages [12026.628940] Lustre: server umount lustre-MDT0000 complete [12030.900363] LustreError: 472386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378584 with bad export cookie 15946003280717627753 [12030.903845] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12030.917386] LustreError: 472386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12031.545485] Lustre: server umount lustre-MDT0001 complete [12050.912070] Lustre: 471662:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789378588/real 1789378588] req@00000000f2bc25f6 x1876298942322048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789378604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12050.949270] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12052.005593] Lustre: 471662:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789378589/real 1789378589] req@00000000fc10ebc5 x1876298942322176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789378605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12055.018084] Lustre: 471664:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789378593/real 1789378593] req@000000006bd139ad x1876298942322304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789378609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12056.942973] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12061.221849] Lustre: 471663:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789378599/real 1789378599] req@000000006d0b1fac x1876298942322688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789378615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12061.267855] Lustre: 471663:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12070.914305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12075.551546] LustreError: 471661:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006c817420 x1876298942323456/t0(0) o250->MGC192.168.201.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12075.849589] LustreError: 474356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12075.878146] LustreError: 474356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12075.955727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12076.968579] LustreError: 474357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12076.988783] LustreError: 474357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12081.089784] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12082.147617] LustreError: 498640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.177526] LustreError: 498640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12084.198781] LustreError: 498641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12084.216869] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12084.230581] LustreError: 498641:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12089.320055] LustreError: 474356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.339266] LustreError: 474356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12089.778454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.068580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12094.351217] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12095.470275] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12103.023460] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12111.104509] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12141.543710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12141.572898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12141.573566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12141.594418] Lustre: Skipped 1 previous similar message [12151.782482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12151.786829] Lustre: Skipped 2 previous similar messages [12152.287182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12152.716202] Lustre: server umount lustre-OST0000 complete [12162.036030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12162.051876] Lustre: Skipped 2 previous similar messages [12162.058278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12163.798067] Lustre: server umount lustre-MDT0000 complete [12167.141297] LustreError: 498645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12167.151163] LustreError: 498645:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12167.201765] LustreError: 472386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378721 with bad export cookie 15946003280717630392 [12167.205852] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12167.221103] LustreError: 472386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12167.530839] Lustre: server umount lustre-MDT0001 complete [12181.604168] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [12185.940919] Key type lgssc unregistered [12186.410135] LNet: 501443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12186.429343] LNetError: 501443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12186.449704] LNet: Removed LNI 192.168.201.104@tcp [12186.458796] LNet: Removed LNI 192.168.201.104@tcp10 [12186.477959] LNet: Removed LNI 192.168.201.104@tcp11 [12186.496537] LNet: Removed LNI 192.168.201.104@tcp12 [12186.505922] LNet: Removed LNI 192.168.201.104@tcp13 [12186.522258] LNet: Removed LNI 192.168.201.104@tcp14 [12186.532715] LNet: Removed LNI 192.168.201.104@tcp15 [12186.538445] LNet: Removed LNI 192.168.201.104@tcp16 [12186.557857] LNet: Removed LNI 192.168.201.104@tcp17 [12186.570712] LNet: Removed LNI 192.168.201.104@tcp18 [12186.597054] LNet: Removed LNI 192.168.201.104@tcp19 [12186.602215] LNet: Removed LNI 192.168.201.104@tcp20 [12186.608589] LNet: Removed LNI 192.168.201.104@tcp21 [12186.617810] LNet: Removed LNI 192.168.201.104@tcp22 [12186.633699] LNet: Removed LNI 192.168.201.104@tcp23 [12186.646390] LNet: Removed LNI 192.168.201.104@tcp24 [12186.660622] LNet: Removed LNI 192.168.201.104@tcp25 [12186.667774] LNet: Removed LNI 192.168.201.104@tcp26 [12186.672755] LNet: Removed LNI 192.168.201.104@tcp27 [12186.685250] LNet: Removed LNI 192.168.201.104@tcp28 [12186.692756] LNet: Removed LNI 192.168.201.104@tcp29 [12186.698808] LNet: Removed LNI 192.168.201.104@tcp30 [12186.713465] LNet: Removed LNI 192.168.201.104@tcp31 [12186.736276] LNet: Removed LNI 192.168.201.104@tcp32 [12186.738347] LNet: Removed LNI 192.168.201.104@tcp33 [12186.752096] LNet: Removed LNI 192.168.201.104@tcp34 [12186.753937] LNet: Removed LNI 192.168.201.104@tcp35 [12186.771125] LNet: Removed LNI 192.168.201.104@tcp36 [12186.773030] LNet: Removed LNI 192.168.201.104@tcp37 [12186.779659] LNet: Removed LNI 192.168.201.104@tcp38 [12186.785540] LNet: Removed LNI 192.168.201.104@tcp39 [12186.794324] LNet: Removed LNI 192.168.201.104@tcp40 [12186.802040] LNet: Removed LNI 192.168.201.104@tcp41 [12186.809253] LNet: Removed LNI 192.168.201.104@tcp42 [12186.812879] LNet: Removed LNI 192.168.201.104@tcp43 [12186.819203] LNet: Removed LNI 192.168.201.104@tcp44 [12186.823925] LNet: Removed LNI 192.168.201.104@tcp45 [12186.835214] LNet: Removed LNI 192.168.201.104@tcp46 [12186.837244] LNet: Removed LNI 192.168.201.104@tcp47 [12186.861446] LNet: Removed LNI 192.168.201.104@tcp48 [12186.863747] LNet: Removed LNI 192.168.201.104@tcp49 [12186.894274] LNet: Removed LNI 192.168.201.104@tcp50 [12188.484135] Key type .llcrypt unregistered [12188.487734] Key type ._llcrypt unregistered [12215.341313] Key type ._llcrypt registered [12215.349925] Key type .llcrypt registered [12215.482336] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [12233.079692] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12234.535193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12234.811718] alg: No test for adler32 (adler32-zlib) [12236.153572] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [12236.480786] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [12238.175210] Key type lgssc registered [12240.028029] Lustre: Echo OBD driver; http://www.lustre.org/ [12281.879584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12281.920766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.242503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12283.288898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12283.378904] Lustre: lustre-MDT0000: new disk, initializing [12283.439837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12283.456748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12289.936076] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12302.831952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12302.992436] Lustre: 504637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12303.082300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12303.085204] Lustre: Skipped 1 previous similar message [12303.202128] Lustre: lustre-MDT0001: new disk, initializing [12303.296142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12303.353265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12303.368812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12308.807778] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12319.943066] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12326.569965] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12334.912686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12335.151359] Lustre: *** cfs_fail_loc=90f, val=0*** [12335.216240] Lustre: lustre-OST0000: new disk, initializing [12335.219078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12335.226465] Lustre: 506654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12335.313404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12342.162774] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12343.429455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12343.451844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12343.618656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12352.543961] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12364.256672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12364.270333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12364.276735] LustreError: Skipped 1 previous similar message [12364.278602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12364.319151] Lustre: Skipped 1 previous similar message [12369.376371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12369.389784] Lustre: Skipped 1 previous similar message [12370.657754] Lustre: server umount lustre-OST0000 complete [12379.619154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12379.633655] Lustre: Skipped 1 previous similar message [12379.638265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12379.644418] Lustre: Skipped 1 previous similar message [12384.737988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12384.740896] Lustre: Skipped 1 previous similar message [12389.866661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12389.882726] Lustre: Skipped 1 previous similar message [12390.368270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12390.663930] Lustre: server umount lustre-MDT0000 complete [12394.914508] LustreError: 504629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378948 with bad export cookie 3560519128837765178 [12394.922123] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12394.929881] LustreError: 504629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12395.533046] Lustre: server umount lustre-MDT0001 complete [12408.413884] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [12411.779466] Key type lgssc unregistered [12412.110380] LNet: 508564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12412.120170] LNetError: 508564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12413.165885] LNet: Removed LNI 192.168.201.104@tcp [12414.051177] Key type .llcrypt unregistered [12414.053695] Key type ._llcrypt unregistered [12434.800376] Key type ._llcrypt registered [12434.802967] Key type .llcrypt registered [12434.908522] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12435.774133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12435.891206] alg: No test for adler32 (adler32-zlib) [12436.951632] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [12437.303732] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [12438.967234] Key type lgssc registered [12440.464679] Lustre: Echo OBD driver; http://www.lustre.org/ [12450.041170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12450.083326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12451.648266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12457.493917] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12461.048311] LustreError: 509829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12461.076134] LustreError: 509829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12466.148835] LustreError: 509830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.424822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12469.748269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12475.300158] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12482.771359] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12490.219483] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12498.831686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12499.489583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12508.731551] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12520.513503] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12536.565042] loop0: detected capacity change from 0 to 204800000 [12536.941031] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12536.970234] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12537.001407] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12546.964780] loop0: detected capacity change from 0 to 204800000 [12547.151737] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12547.168755] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12547.195627] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12554.804739] loop0: detected capacity change from 0 to 204800000 [12561.433759] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12561.646470] Lustre: fs15246-OST0000: new disk, initializing [12561.652952] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12561.666995] Lustre: 513424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12561.831157] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12568.863881] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12580.899185] loop1: detected capacity change from 0 to 204800000 [12587.741039] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12587.832291] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12587.886228] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12587.972155] Lustre: fs15246-MDT0000: new disk, initializing [12588.248229] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12588.285362] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12588.429716] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12589.487562] Lustre: 514546:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002f712147 x1876299864920448/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789379159 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12594.109497] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12598.627212] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12598.635182] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12598.690121] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12620.287528] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12620.325779] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12624.340565] Lustre: server umount fs15246-MDT0000 complete [12625.388066] LustreError: 509826:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12630.530709] LustreError: 509826:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12635.259650] Lustre: server umount fs15246-OST0000 complete [12650.978897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12651.011954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12655.891174] Lustre: server umount lustre-OST0000 complete [12661.218692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12661.222648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12661.238920] Lustre: Skipped 2 previous similar messages [12661.248715] Lustre: Skipped 2 previous similar messages [12665.697713] Lustre: server umount lustre-MDT0000 complete [12666.339870] LustreError: 509830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12666.368483] LustreError: 509830:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12669.431366] LustreError: 509810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789379223 with bad export cookie 13401450022711259078 [12669.437513] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12669.757112] Lustre: server umount lustre-MDT0001 complete [12680.561623] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [12683.628453] Key type lgssc unregistered [12684.013243] LNet: 517647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12684.026701] LNetError: 517647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12684.049945] LNet: Removed LNI 192.168.201.104@tcp [12685.186740] Key type .llcrypt unregistered [12685.193300] Key type ._llcrypt unregistered [12708.289877] Key type ._llcrypt registered [12708.301707] Key type .llcrypt registered [12708.495799] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12710.290337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12710.352069] alg: No test for adler32 (adler32-zlib) [12711.584258] Lustre: Lustre: Build Version: 2.17.58_39_g3ab1dfb [12711.981162] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [12713.687316] Key type lgssc registered [12714.733926] Lustre: Echo OBD driver; http://www.lustre.org/ [12757.691994] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [12766.933919] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12804.460779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12804.508819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12806.010355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12806.084968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12806.167938] Lustre: lustre-MDT0000: new disk, initializing [12806.289264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12806.322985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12810.817215] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12822.867042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12823.078477] Lustre: 522113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12823.219987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12823.228603] Lustre: Skipped 1 previous similar message [12823.380317] Lustre: lustre-MDT0001: new disk, initializing [12823.488562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12823.533019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12823.546582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12828.448257] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12839.329907] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12845.118520] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12852.195117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12852.745603] Lustre: lustre-OST0000: new disk, initializing [12852.749310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12852.754775] Lustre: 524082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12852.894203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12859.994817] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12862.607513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12862.626354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12862.802287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12870.029555] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12877.045771] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12877.454428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12883.985860] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12884.370042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12888.548507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12888.554971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12888.584738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12888.587421] Lustre: Skipped 1 previous similar message [12893.666820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12893.685355] Lustre: Skipped 1 previous similar message [12898.804746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12898.820927] Lustre: Skipped 1 previous similar message [12901.343131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12901.630722] Lustre: server umount lustre-OST0000 complete [12909.028371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12909.037069] Lustre: Skipped 1 previous similar message [12909.043037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12909.048543] Lustre: Skipped 1 previous similar message [12914.144439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12914.148558] Lustre: Skipped 1 previous similar message [12921.311136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12921.763530] Lustre: server umount lustre-MDT0000 complete [12925.899651] LustreError: 522104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789379479 with bad export cookie 10374041990566286545 [12925.922281] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12926.433776] Lustre: server umount lustre-MDT0001 complete [12953.508748] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [12972.052122] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [13009.230192] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12697 sec ========= 05:52:41 (1789379561) [13011.360051] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:52:42 (1789379562) === [13015.591314] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:52:47 (1789379567) === [13064.312551] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13068.721772] Key type lgssc unregistered [13069.393762] LNet: 530754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13069.407733] LNetError: 530754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13069.451819] LNet: Removed LNI 192.168.201.104@tcp [13070.832520] Key type .llcrypt unregistered [13070.836119] Key type ._llcrypt unregistered