[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 510371079 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2512MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002316] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009123] LSM: Security Framework initializing [ 0.010041] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012070] *** VALIDATE selinux *** [ 0.020008] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026241] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027211] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028154] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029170] *** VALIDATE tmpfs *** [ 0.030605] *** VALIDATE proc *** [ 0.031349] *** VALIDATE cgroup *** [ 0.032014] *** VALIDATE cgroup2 *** [ 0.034319] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035194] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037065] Spectre V2 : User space: Vulnerable [ 0.038029] Speculative Store Bypass: Vulnerable [ 0.042522] debug: unmapping init [mem 0xffffffff97059000-0xffffffff97060fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045671] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.047988] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050011] ... fixed-purpose events: 3 [ 0.050929] ... event mask: 000000070000000f [ 0.051293] rcu: Hierarchical SRCU implementation. [ 0.053419] smp: Bringing up secondary CPUs ... [ 0.054533] x86: Booting SMP configuration: [ 0.055034] .... node #0, CPUs: #1 #2 #3 [ 0.058488] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.060991] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.231532] node 0 deferred pages initialised in 168ms [ 0.235182] devtmpfs: initialized [ 0.236264] x86/mm: Memory block size: 128MB [ 0.238502] gcov: version magic: 0x41383552 [ 0.240434] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246432] pinctrl core: initialized pinctrl subsystem [ 0.248263] [ 0.248832] ************************************************************* [ 0.251020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253014] ** ** [ 0.256017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.258016] ** ** [ 0.261020] ** This means that this kernel is built to expose internal ** [ 0.263018] ** IOMMU data structures, which may compromise security on ** [ 0.265017] ** your system. ** [ 0.267021] ** ** [ 0.270060] ** If you see this message and you are not debugging the ** [ 0.273019] ** kernel, report this immediately to your vendor! ** [ 0.275016] ** ** [ 0.277021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279020] ************************************************************* [ 0.282886] NET: Registered protocol family 16 [ 0.285496] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.288084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.291090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.295041] cpuidle: using governor menu [ 0.299316] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.302840] PCI: Using configuration type 1 for base access [ 0.304387] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.315258] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.316021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.318156] cryptd: max_cpu_qlen set to 1000 [ 0.320249] ACPI: Added _OSI(Module Device) [ 0.322042] ACPI: Added _OSI(Processor Device) [ 0.323000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.323000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.328243] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.335765] ACPI: Interpreter enabled [ 0.337063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338011] ACPI: Using IOAPIC for interrupt routing [ 0.339105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.342391] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.353921] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.355048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.358032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.362150] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.367614] acpiphp: Slot [2] registered [ 0.369191] acpiphp: Slot [5] registered [ 0.373247] acpiphp: Slot [6] registered [ 0.374000] acpiphp: Slot [7] registered [ 0.374000] acpiphp: Slot [8] registered [ 0.376191] acpiphp: Slot [9] registered [ 0.377184] acpiphp: Slot [10] registered [ 0.379243] acpiphp: Slot [3] registered [ 0.381125] acpiphp: Slot [4] registered [ 0.382145] acpiphp: Slot [11] registered [ 0.384151] acpiphp: Slot [12] registered [ 0.385160] acpiphp: Slot [13] registered [ 0.386127] acpiphp: Slot [14] registered [ 0.388161] acpiphp: Slot [15] registered [ 0.389132] acpiphp: Slot [16] registered [ 0.391147] acpiphp: Slot [17] registered [ 0.392138] acpiphp: Slot [18] registered [ 0.394140] acpiphp: Slot [19] registered [ 0.396180] acpiphp: Slot [20] registered [ 0.397162] acpiphp: Slot [21] registered [ 0.399136] acpiphp: Slot [22] registered [ 0.400133] acpiphp: Slot [23] registered [ 0.402133] acpiphp: Slot [24] registered [ 0.403145] acpiphp: Slot [25] registered [ 0.405126] acpiphp: Slot [26] registered [ 0.406134] acpiphp: Slot [27] registered [ 0.408173] acpiphp: Slot [28] registered [ 0.409132] acpiphp: Slot [29] registered [ 0.411125] acpiphp: Slot [30] registered [ 0.412136] acpiphp: Slot [31] registered [ 0.414085] PCI host bridge to bus 0000:00 [ 0.415031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.418041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.420037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.423040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.426049] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.428054] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.430236] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.434157] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.437406] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.447779] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.452516] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.455030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.458042] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.460031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.463639] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.465931] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.468047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.471038] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.477021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.490018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.496014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.502384] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.512025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.522027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.547029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.557326] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.566081] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.578026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.594019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.605000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.614026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.622028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.641028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.651328] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.662025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.672027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.691020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.700458] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.715025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.725024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.748027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.762000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.767018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.773016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.805024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.816724] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.819452] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.822479] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.828639] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.831277] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.836326] iommu: Default domain type: Passthrough [ 0.839537] SCSI subsystem initialized [ 0.841219] ACPI: bus type USB registered [ 0.842153] usbcore: registered new interface driver usbfs [ 0.845153] usbcore: registered new interface driver hub [ 0.847134] usbcore: registered new device driver usb [ 0.849216] pps_core: LinuxPPS API ver. 1 registered [ 0.850012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.854106] PTP clock support registered [ 0.856101] EDAC MC: Ver: 3.0.0 [ 0.857546] PCI: Using ACPI for IRQ routing [ 0.858663] NetLabel: Initializing [ 0.860017] NetLabel: domain hash size = 128 [ 0.861014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.863087] NetLabel: unlabeled traffic allowed by default [ 0.865156] vgaarb: loaded [ 0.867239] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.869022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.875391] clocksource: Switched to clocksource kvm-clock [ 0.987772] VFS: Disk quotas dquot_6.6.0 [ 0.989598] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.992141] *** VALIDATE ramfs *** [ 0.993507] *** VALIDATE hugetlbfs *** [ 0.995373] pnp: PnP ACPI init [ 0.998091] pnp: PnP ACPI: found 6 devices [ 1.024729] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.028563] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.030895] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.032787] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.035778] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.038385] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.041200] NET: Registered protocol family 2 [ 1.043851] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.049368] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.053561] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.059104] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.062542] TCP: Hash tables configured (established 65536 bind 65536) [ 1.065510] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.068693] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.071403] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.074324] NET: Registered protocol family 1 [ 1.077620] RPC: Registered named UNIX socket transport module. [ 1.079818] RPC: Registered udp transport module. [ 1.080995] RPC: Registered tcp transport module. [ 1.082242] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.084721] NET: Registered protocol family 44 [ 1.086729] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.088953] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.091277] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.093028] PCI: CLS 0 bytes, default 64 [ 1.094690] Unpacking initramfs... [ 2.572917] debug: unmapping init [mem 0xffff995f3cc54000-0xffff995f3ffbffff] [ 2.579724] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.582216] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.586690] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.111146] Initialise system trusted keyrings [ 3.113064] Key type blacklist registered [ 3.115437] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.125149] zbud: loaded [ 3.128315] *** VALIDATE nfs *** [ 3.129681] *** VALIDATE nfs4 *** [ 3.131514] pstore: using deflate compression [ 3.135166] Platform Keyring initialized [ 3.244682] NET: Registered protocol family 38 [ 3.246170] Key type asymmetric registered [ 3.247800] Asymmetric key parser 'x509' registered [ 3.250082] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.253725] io scheduler mq-deadline registered [ 3.255500] io scheduler kyber registered [ 3.257012] io scheduler bfq registered [ 3.258814] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.261600] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.264560] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.267478] ACPI: Power Button [PWRF] [ 3.272976] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.279612] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.293475] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.309856] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.328588] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.357300] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.385103] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.390337] Non-volatile memory driver v1.3 [ 3.392027] Linux agpgart interface v0.103 [ 3.427995] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.431025] vda: detected capacity change from 0 to 76677120 [ 3.450754] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.453947] vdb: detected capacity change from 0 to 1073741824 [ 3.481419] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.487277] vdc: detected capacity change from 0 to 2621440000 [ 3.503584] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.506924] vdd: detected capacity change from 0 to 2621440000 [ 3.520780] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.523223] vde: detected capacity change from 0 to 4294967296 [ 3.540904] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.543928] vdf: detected capacity change from 0 to 4294967296 [ 3.555259] libphy: Fixed MDIO Bus: probed [ 3.560267] usbcore: registered new interface driver usbserial_generic [ 3.562801] usbserial: USB Serial support registered for generic [ 3.565100] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.569424] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.570926] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.573668] mousedev: PS/2 mouse device common for all mice [ 3.576497] rtc_cmos 00:05: RTC can wake from S4 [ 3.580695] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.582377] rtc_cmos 00:05: registered as rtc0 [ 3.587086] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.587102] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.590717] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.592697] intel_pstate: CPU model not supported [ 3.598834] hid: raw HID events driver (C) Jiri Kosina [ 3.601415] usbcore: registered new interface driver usbhid [ 3.604016] usbhid: USB HID core driver [ 3.606333] drop_monitor: Initializing network drop monitor service [ 3.608652] Initializing XFRM netlink socket [ 3.610638] NET: Registered protocol family 10 [ 3.614186] Segment Routing with IPv6 [ 3.615614] NET: Registered protocol family 17 [ 3.617912] mpls_gso: MPLS GSO support [ 3.624325] RAS: Correctable Errors collector initialized. [ 3.626502] AVX version of gcm_enc/dec engaged. [ 3.628308] AES CTR mode by8 optimization enabled [ 3.720552] sched_clock: Marking stable (3720520167, 0)->(4621075675, -900555508) [ 3.723537] registered taskstats version 1 [ 3.725339] Loading compiled-in X.509 certificates [ 3.726888] zswap: loaded using pool lzo/zbud [ 3.762901] Key type big_key registered [ 3.774559] Key type encrypted registered [ 3.775814] ima: No TPM chip found, activating TPM-bypass! [ 3.777353] ima: Allocated hash algorithm: sha1 [ 3.778887] ima: No architecture policies found [ 3.780197] evm: Initialising EVM extended attributes: [ 3.781585] evm: security.selinux [ 3.782506] evm: security.ima [ 3.783319] evm: security.capability [ 3.784277] evm: HMAC attrs: 0x1 [ 3.786303] rtc_cmos 00:05: setting system clock to 2026-09-05 15:14:51 UTC (1788621291) [ 3.791976] debug: unmapping init [mem 0xffffffff98003000-0xffffffff981fffff] [ 3.794672] debug: unmapping init [mem 0xffffffff96d82000-0xffffffff97058fff] [ 3.803239] Write protecting the kernel read-only data: 28672k [ 3.806599] debug: unmapping init [mem 0xffffffff95403000-0xffffffff955fffff] [ 3.809182] debug: unmapping init [mem 0xffffffff95d14000-0xffffffff95dfffff] [ 3.850742] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.857590] systemd[1]: Detected virtualization kvm. [ 3.859448] systemd[1]: Detected architecture x86-64. [ 3.861824] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.889751] systemd[1]: No hostname configured. [ 3.891494] systemd[1]: Set hostname to . [ 3.893711] random: systemd: uninitialized urandom read (16 bytes read) [ 3.895803] systemd[1]: Initializing machine ID from random generator. [ 4.052981] random: systemd: uninitialized urandom read (16 bytes read) [ 4.055441] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.060245] random: systemd: uninitialized urandom read (16 bytes read) [ 4.062669] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.068504] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.681696] device-mapper: uevent: version 1.0.3 [ 4.684106] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.367049] random: fast init done [ 5.375621] virtio_net virtio0 ens2: renamed from eth0 [ 5.422115] scsi host0: ata_piix [ 5.432883] scsi host1: ata_piix [ 5.434444] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.437110] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.735364] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.288229] random: crng init done [ 10.289601] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.677679] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.835521] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.104176] SELinux: Disabled at runtime. [ 12.169438] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.178326] systemd[1]: Detected virtualization kvm. [ 12.180296] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.718484] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.721957] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.726722] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.730574] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.737149] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.747189] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.754133] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.937834] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.255199] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.523697] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.657910] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.804090] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.814746] EDAC sbridge: Ver: 1.1.2 [ 15.667146] Key type dns_resolver registered [ 15.975815] NFS: Registering the id_resolver key type [ 15.977879] Key type id_resolver registered [ 15.979603] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg429-server login: [ 42.305982] libcfs: loading out-of-tree module taints kernel. [ 42.329093] Key type ._llcrypt registered [ 42.330537] Key type .llcrypt registered [ 42.388129] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 50.916700] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 51.907468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.944692] alg: No test for adler32 (adler32-zlib) [ 53.369931] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 54.553521] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 56.447233] Key type lgssc registered [ 58.772171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 78.113905] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 110.511073] hrtimer: interrupt took 4061059 ns [ 121.565526] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 135.518339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 135.558347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 136.868752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 136.900309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 137.081838] Lustre: lustre-MDT0000: new disk, initializing [ 137.229568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.262324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 141.903173] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.803759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.961300] Lustre: 6492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.013929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.019325] Lustre: Skipped 1 previous similar message [ 158.113715] Lustre: lustre-MDT0001: new disk, initializing [ 158.251059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.296048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.310584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.731285] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.554853] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.306140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.587753] Lustre: lustre-OST0000: new disk, initializing [ 180.597657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.606982] Lustre: 8428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.723219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.938034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 181.953265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 182.057844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 186.760405] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.341372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.510990] Lustre: lustre-OST0001: new disk, initializing [ 201.514441] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 201.523606] Lustre: 9499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.592080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.108000] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.513802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 209.524886] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 209.632242] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.861258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 230.941172] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 237.499808] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.462526] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing yml_node [ 248.077151] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 250.730189] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 254.429454] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 256.377034] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 11:19:01 EDT 2026 [ 277.697635] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 279.419640] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 296.417922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 296.423786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.439608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.929830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.946336] Lustre: Skipped 2 previous similar messages [ 296.953192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.959116] Lustre: Skipped 2 previous similar messages [ 301.298176] Lustre: server umount lustre-MDT0000 complete [ 304.385541] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621592 with bad export cookie 9606036982263299892 [ 304.396236] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.181256] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.182878] LustreError: 6498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.183635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.204337] Lustre: Skipped 1 previous similar message [ 307.227283] LustreError: 6498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 309.216509] LustreError: 6498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.221345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.254938] Lustre: Skipped 2 previous similar messages [ 310.918651] Lustre: server umount lustre-MDT0001 complete [ 314.684731] Lustre: server umount lustre-OST0000 complete [ 318.496969] Lustre: server umount lustre-OST0001 complete [ 326.133504] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 335.746610] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 378.560610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.954652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 379.009730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 379.163774] Lustre: lustre-MDT0000: new disk, initializing [ 379.267070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 379.327437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.978526] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 400.748776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.039323] Lustre: 15054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 401.158182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 401.167444] Lustre: Skipped 1 previous similar message [ 401.326643] Lustre: lustre-MDT0001: new disk, initializing [ 401.516446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 401.601897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 401.618315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 406.805860] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 418.899921] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 426.061618] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 433.588276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.958349] Lustre: lustre-OST0000: new disk, initializing [ 433.962522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 433.968702] Lustre: 17020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.056986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.145215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.173144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.332215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 441.308544] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 451.818749] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 456.876983] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 457.139468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 461.520299] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.701757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.889225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.892218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.910195] LustreError: Skipped 1 previous similar message [ 465.924789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.928600] Lustre: Skipped 1 previous similar message [ 465.965106] Lustre: Skipped 1 previous similar message [ 476.133726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.139054] Lustre: Skipped 3 previous similar messages [ 478.177540] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 478.308853] Lustre: server umount lustre-OST0000 complete [ 483.297605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 483.305425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.096069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 496.610273] LustreError: 15065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.636303] LustreError: 15065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 496.695302] Lustre: server umount lustre-MDT0000 complete [ 500.734633] LustreError: 15047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621788 with bad export cookie 9606036982263301425 [ 500.751225] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.759277] LustreError: 15047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 501.096258] Lustre: server umount lustre-MDT0001 complete [ 505.798971] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:23:11 (1788621791) [ 508.464324] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 511.106526] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:23:16 (1788621796) [ 518.037320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.614919] LustreError: 19436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.711718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.747599] LustreError: 19437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.430919] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 533.989103] LustreError: 19436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.023428] LustreError: 19436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 535.338191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.373927] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 548.763766] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 555.207821] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 562.109858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 569.935990] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 579.273273] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 594.400835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.410681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.425123] Lustre: Skipped 1 previous similar message [ 594.434937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 594.441298] Lustre: Skipped 5 previous similar messages [ 598.977589] Lustre: server umount lustre-OST0000 complete [ 609.283832] Lustre: server umount lustre-MDT0000 complete [ 611.809716] LustreError: 19437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.845253] LustreError: 19437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 614.129042] LustreError: 22238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621901 with bad export cookie 9606036982263302307 [ 614.140772] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.148061] LustreError: 22238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 614.717000] Lustre: server umount lustre-MDT0001 complete [ 627.234334] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 630.247785] Key type lgssc unregistered [ 630.629949] LNet: 22990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.647422] LNetError: 22990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.674288] LNet: Removed LNI 192.168.204.129@tcp [ 631.916447] Key type .llcrypt unregistered [ 631.924289] Key type ._llcrypt unregistered [ 657.946894] Key type ._llcrypt registered [ 657.949479] Key type .llcrypt registered [ 658.136944] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 659.483681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.682265] alg: No test for adler32 (adler32-zlib) [ 661.276153] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 661.884668] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 663.735827] Key type lgssc registered [ 665.693447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 678.368086] loop: module loaded [ 678.484251] loop0: detected capacity change from 0 to 204800000 [ 678.948367] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.973731] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.863307] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.070538] loop0: detected capacity change from 0 to 204800000 [ 687.494781] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.531484] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.545527] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.442953] loop0: detected capacity change from 0 to 204800000 [ 701.470700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 701.512362] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.699114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 702.743679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 702.832100] Lustre: lustre-MDT0000: new disk, initializing [ 702.935512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.956299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 708.736322] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 719.609277] loop1: detected capacity change from 0 to 204800000 [ 726.047637] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 726.378391] Lustre: lustre-OST07c6: new disk, initializing [ 726.382852] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 726.392356] Lustre: Skipped 1 previous similar message [ 726.403364] Lustre: 26174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 726.496961] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 733.812527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 733.829055] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 733.858063] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 734.452803] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 740.063714] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 740.083690] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 740.091203] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 740.102537] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 740.121455] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 740.133565] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 760.371796] Lustre: 25161:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 760.388975] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.398266] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.415595] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 760.425686] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 760.434095] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.421760] Lustre: 25161:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 765.434598] Lustre: 25161:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 765.441538] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 765.458222] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.464797] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 765.474731] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.484174] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 765.500335] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.511077] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 765.517519] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.530976] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.537152] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.784723] Lustre: 25161:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 768.798500] Lustre: 25161:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 768.802774] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 768.813692] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.824357] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.830519] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.838510] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 768.845977] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.854299] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 768.862898] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.874390] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.883650] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.929803] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 773.942828] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 773.954033] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 773.965818] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.975076] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 773.980988] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.992087] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 774.005362] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.018662] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 774.032352] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.040695] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 774.048757] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.246970] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 782.259868] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 782.266941] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 782.274521] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 782.282190] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 782.290496] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 782.296505] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 782.303817] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 782.309539] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 782.315787] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 782.322886] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 782.332029] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 841.187210] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.205553] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 843.209341] Lustre: server umount lustre-OST07c6 complete [ 854.179875] Lustre: server umount lustre-MDT0000 complete [ 873.534380] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 877.467477] Key type lgssc unregistered [ 877.740703] LNet: 28732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.754233] LNetError: 28732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.772619] LNet: Removed LNI 192.168.204.129@tcp [ 879.081850] Key type .llcrypt unregistered [ 879.087193] Key type ._llcrypt unregistered [ 900.805460] Key type ._llcrypt registered [ 900.818892] Key type .llcrypt registered [ 901.012603] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 902.246072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 902.314197] alg: No test for adler32 (adler32-zlib) [ 903.552146] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 903.971789] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 905.671193] Key type lgssc registered [ 906.950202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.723556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.757084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.375874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.434364] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 927.713057] LustreError: 29998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.744554] LustreError: 29998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 930.424919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.903525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.737847] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 944.806734] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 951.632755] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 959.571949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 959.907712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.001239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 967.355542] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 976.752320] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 996.861666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 996.865943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.892413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1001.757907] Lustre: server umount lustre-OST0000 complete [ 1007.073694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.101167] Lustre: Skipped 1 previous similar message [ 1007.106791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.115495] Lustre: Skipped 2 previous similar messages [ 1012.196143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1012.205436] Lustre: Skipped 1 previous similar message [ 1016.289176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.300117] Lustre: Skipped 1 previous similar message [ 1020.385059] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1020.734369] Lustre: server umount lustre-MDT0000 complete [ 1021.408775] LustreError: 29993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.434323] LustreError: 29993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1024.839892] LustreError: 29981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622312 with bad export cookie 9461854483058329674 [ 1024.841222] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1024.857876] LustreError: 29981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1025.311863] Lustre: server umount lustre-MDT0001 complete [ 1036.973388] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1040.641832] Key type lgssc unregistered [ 1041.059824] LNet: 33738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.070069] LNetError: 33738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.100422] LNet: Removed LNI 192.168.204.129@tcp [ 1042.280622] Key type .llcrypt unregistered [ 1042.285220] Key type ._llcrypt unregistered [ 1065.362872] Key type ._llcrypt registered [ 1065.364942] Key type .llcrypt registered [ 1065.504329] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1066.253026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1066.345172] alg: No test for adler32 (adler32-zlib) [ 1067.616776] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 1067.942984] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1069.712751] Key type lgssc registered [ 1070.856817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1079.456224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1079.477405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.051522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.414359] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1090.529503] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1090.543936] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1095.649037] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.448339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.940389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1102.025552] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1111.157622] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1118.002147] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1125.738850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1126.229397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.555894] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1142.803398] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1150.878648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1166.306805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.309067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1166.333089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1171.427058] LustreError: 36828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.695393] Lustre: server umount lustre-OST0000 complete [ 1176.547293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.572986] Lustre: Skipped 1 previous similar message [ 1176.578646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1176.585679] Lustre: Skipped 2 previous similar messages [ 1181.669028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1181.691990] Lustre: Skipped 1 previous similar message [ 1181.948058] Lustre: server umount lustre-MDT0000 complete [ 1186.501624] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622474 with bad export cookie 9331649210919433498 [ 1186.506176] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.510233] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1186.933565] Lustre: server umount lustre-MDT0001 complete [ 1198.747732] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1201.763242] Key type lgssc unregistered [ 1202.002315] LNet: 38564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.009604] LNetError: 38564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.032463] LNet: Removed LNI 192.168.204.129@tcp [ 1202.892757] Key type .llcrypt unregistered [ 1202.900210] Key type ._llcrypt unregistered [ 1228.363756] Key type ._llcrypt registered [ 1228.365757] Key type .llcrypt registered [ 1228.456397] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1229.876337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1229.901345] alg: No test for adler32 (adler32-zlib) [ 1231.109974] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 1231.511831] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1233.287316] Key type lgssc registered [ 1234.459120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.315808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.338541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1243.740335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.731961] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1253.864236] LustreError: 39975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.877384] LustreError: 39975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1258.197945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.566732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1263.337290] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1270.559960] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1276.588895] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1284.425784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.836581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.954213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1291.494958] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1301.377835] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1309.664365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1309.671145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.684138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1309.699865] Lustre: Skipped 1 previous similar message [ 1309.709787] Lustre: Skipped 2 previous similar messages [ 1314.808539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1314.818797] Lustre: Skipped 3 previous similar messages [ 1315.867237] Lustre: server umount lustre-MDT0000 complete [ 1319.920304] LustreError: 39971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.933568] LustreError: 39971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1320.009544] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622607 with bad export cookie 10394039036773130538 [ 1320.024914] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1320.026229] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.016027] LustreError: 40705:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007767831c x1875506646511488/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1325.024911] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.025453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.053617] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1325.077526] Lustre: Skipped 1 previous similar message [ 1325.085313] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1325.087815] Lustre: Skipped 1 previous similar message [ 1326.573532] Lustre: server umount lustre-MDT0001 complete [ 1348.821622] Lustre: server umount lustre-OST0000 complete [ 1368.924601] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1372.632078] Key type lgssc unregistered [ 1373.029753] LNet: 43840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.039965] LNetError: 43840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.075335] LNet: Removed LNI 192.168.204.129@tcp [ 1373.922593] Key type .llcrypt unregistered [ 1373.926531] Key type ._llcrypt unregistered [ 1404.059959] Key type ._llcrypt registered [ 1404.068687] Key type .llcrypt registered [ 1404.192914] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1405.148253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1405.214716] alg: No test for adler32 (adler32-zlib) [ 1406.321211] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 1406.649536] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1408.520065] Key type lgssc registered [ 1410.244563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1419.103481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1419.125993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.554132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.550062] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1431.016965] LustreError: 45255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.052447] LustreError: 45255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1436.017211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.129533] LustreError: 45254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.592114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1440.857532] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1448.427653] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1455.212940] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1462.323205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1462.700739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.780634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1468.977616] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1476.551608] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1487.328373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1487.338355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.341624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1487.364813] Lustre: Skipped 1 previous similar message [ 1488.944657] Lustre: server umount lustre-OST0000 complete [ 1507.694041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.721742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.733899] Lustre: Skipped 2 previous similar messages [ 1508.770814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.781757] Lustre: Skipped 1 previous similar message [ 1513.960492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1513.971361] Lustre: Skipped 1 previous similar message [ 1514.161518] Lustre: server umount lustre-MDT0000 complete [ 1518.516956] LustreError: 45236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622806 with bad export cookie 17055126611374845742 [ 1518.521572] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1518.540045] LustreError: 45236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1525.015374] Lustre: server umount lustre-MDT0001 complete [ 1539.093209] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1542.597822] Key type lgssc unregistered [ 1542.951796] LNet: 48967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.957510] LNetError: 48967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.971592] LNet: Removed LNI 192.168.204.129@tcp [ 1544.020399] Key type .llcrypt unregistered [ 1544.023362] Key type ._llcrypt unregistered [ 1569.692479] Key type ._llcrypt registered [ 1569.694880] Key type .llcrypt registered [ 1569.812720] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1570.781472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1570.839390] alg: No test for adler32 (adler32-zlib) [ 1571.948902] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 1572.220655] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1573.903327] Key type lgssc registered [ 1575.202422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1584.582345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1584.601678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.118149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.015622] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1595.361732] LustreError: 50386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.386548] LustreError: 50386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1600.484781] LustreError: 50385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1600.571842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.001170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1606.083587] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1614.673133] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1621.475193] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1628.538085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1628.849577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.907530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1634.766647] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1642.859579] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1652.622988] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1656.648839] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1668.999914] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0511h42m33s [ 1670.521986] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1672.682645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1672.693355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.711382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.763675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.770222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.794752] Lustre: Skipped 1 previous similar message [ 1674.851916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.864375] Lustre: Skipped 3 previous similar messages [ 1676.908114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1676.917406] Lustre: Skipped 3 previous similar messages [ 1678.634974] Lustre: server umount lustre-MDT0000 complete [ 1678.959855] LustreError: 50380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.973561] LustreError: 50380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1680.997052] LustreError: 51368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.024607] LustreError: 51368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1682.381756] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1685.100056] LustreError: 51368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.119030] LustreError: 51368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 1689.528546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.662492] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1689.945726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.996614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1692.007284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1694.222203] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1697.662692] Lustre: DEBUG MARKER: Wait for df (36682) ... [ 1699.433338] Lustre: DEBUG MARKER: done [ 1705.441386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1705.451225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1705.475236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1705.485553] Lustre: Skipped 2 previous similar messages [ 1707.492664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.735659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1717.750640] Lustre: Skipped 5 previous similar messages [ 1719.263224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1719.482592] Lustre: server umount lustre-OST0000 complete [ 1726.945552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1726.956404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.115653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.125560] Lustre: Skipped 5 previous similar messages [ 1738.207172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1738.355138] Lustre: server umount lustre-MDT0000 complete [ 1740.259725] LustreError: 50386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.284726] LustreError: 50386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 1742.217151] LustreError: 53014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623029 with bad export cookie 7452087132082607137 [ 1742.218255] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1742.229022] LustreError: 53014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1742.578409] Lustre: server umount lustre-MDT0001 complete [ 1754.791972] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1757.773887] Key type lgssc unregistered [ 1758.084464] LNet: 55313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.101588] LNetError: 55313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.121908] LNet: Removed LNI 192.168.204.129@tcp [ 1759.018140] Key type .llcrypt unregistered [ 1759.020327] Key type ._llcrypt unregistered [ 1803.141218] Key type ._llcrypt registered [ 1803.144523] Key type .llcrypt registered [ 1803.297729] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1804.425562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.487932] alg: No test for adler32 (adler32-zlib) [ 1805.588159] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 1805.829038] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1807.503257] Key type lgssc registered [ 1808.935278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1816.866273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1816.900956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.259219] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1818.275933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1818.468263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.828947] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1831.076312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.209645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1831.275176] Lustre: 57515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1831.554669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1836.013238] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1843.747597] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1850.464797] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1857.807968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1858.047200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1858.298104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.798427] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1865.880200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1872.841416] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1880.473617] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0511h46m05s [ 1882.478725] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1891.701352] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1894.677119] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1894.756742] LustreError: 59117:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1914.934200] Lustre: lustre-MDT0000: Client 2dd86b13-9e80-4995-b7b5-d28a8d811fb2 (at 192.168.204.29@tcp) reconnecting [ 1934.848424] LustreError: 59117:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1937.025722] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1939.714487] Lustre: DEBUG MARKER: done [ 1947.618835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1947.631194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.664807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1948.133206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1948.140462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.120530] Lustre: server umount lustre-OST0000 complete [ 1958.371936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.391552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.404779] Lustre: Skipped 3 previous similar messages [ 1959.460751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.472996] Lustre: Skipped 1 previous similar message [ 1963.628278] Lustre: server umount lustre-MDT0000 complete [ 1968.213330] LustreError: 58257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623255 with bad export cookie 16771489394707957494 [ 1968.220376] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.227368] LustreError: 58257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1968.697600] Lustre: server umount lustre-MDT0001 complete [ 1980.273719] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1983.242311] Key type lgssc unregistered [ 1983.576594] LNet: 61846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.586209] LNetError: 61846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.610639] LNet: Removed LNI 192.168.204.129@tcp [ 1984.533981] Key type .llcrypt unregistered [ 1984.540676] Key type ._llcrypt unregistered [ 2031.077885] Key type ._llcrypt registered [ 2031.079694] Key type .llcrypt registered [ 2031.220590] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2032.418188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2032.491986] alg: No test for adler32 (adler32-zlib) [ 2033.655188] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 2034.075359] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2035.808315] Key type lgssc registered [ 2036.721732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.604163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.620475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.954913] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2041.997521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2042.187192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.366489] Lustre: Failing over lustre-MDT0000 [ 2043.949593] Lustre: server umount lustre-MDT0000 complete [ 2051.258971] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:48:56 (1788623336) [ 2058.459917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.881379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.620211] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2072.862578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.969450] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2072.996588] Lustre: 64642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2073.186703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.120524] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2086.597931] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2092.479498] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2100.644356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2100.854856] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2101.109696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.659173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2110.745713] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2121.241614] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2128.860911] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2139.620422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2139.622679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.627332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.659437] Lustre: Skipped 2 previous similar messages [ 2141.755297] Lustre: server umount lustre-MDT0000 complete [ 2145.670054] LustreError: 64632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623433 with bad export cookie 8885108220265587710 [ 2145.673717] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.683059] LustreError: 64632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2145.968570] Lustre: server umount lustre-MDT0001 complete [ 2149.720096] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2154.856405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2165.105631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2166.244043] Lustre: 63562:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788623437/real 1788623437] req@000000002dda291a x1875507487484288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788623453 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2166.286125] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.328536] Lustre: 63562:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788623438/real 1788623438] req@000000009c1f0829 x1875507487484416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788623454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2170.335221] Lustre: 63565:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788623442/real 1788623442] req@000000007c69f334 x1875507487484544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788623458 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2170.909692] LustreError: 66489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.931797] LustreError: 66489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2171.051829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.198653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2174.049791] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.097576] LustreError: 66490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2176.264271] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2177.120120] Lustre: 63562:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788623448/real 1788623448] req@00000000131e39fa x1875507487484928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788623464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.128058] LustreError: 68414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.152921] Lustre: 63562:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2177.175020] LustreError: 68414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2182.240893] Lustre: 63563:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788623453/real 1788623453] req@00000000d3dd4de8 x1875507487485056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788623469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2182.627707] LustreError: 68412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.652230] LustreError: 68412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2185.572267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.062881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2190.727449] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2191.350177] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2194.417975] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2200.256486] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2208.697904] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2220.012033] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.019267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2220.044330] Lustre: Skipped 1 previous similar message [ 2220.067388] Lustre: Skipped 4 previous similar messages [ 2225.118579] Lustre: server umount lustre-MDT0000 complete [ 2225.125402] LustreError: 68412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.143100] LustreError: 68412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2229.895567] LustreError: 64634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623517 with bad export cookie 8885108220265589985 [ 2229.904314] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2229.911235] LustreError: 64634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2230.245634] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.265293] Lustre: Skipped 1 previous similar message [ 2230.456308] Lustre: server umount lustre-MDT0001 complete [ 2237.623562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2248.537064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.815577] LustreError: 63561:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000856ac92a x1875507487516160/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2255.137604] LustreError: 66489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.153341] LustreError: 66489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2255.222133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.271473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2259.312584] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2259.953404] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2270.183302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.469107] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2275.835112] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2283.725252] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.011658] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2301.410475] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.416352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2301.422166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2301.434546] Lustre: Skipped 2 previous similar messages [ 2306.539847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2306.545446] Lustre: Skipped 4 previous similar messages [ 2311.651105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.659652] Lustre: Skipped 2 previous similar messages [ 2313.183630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2313.350320] Lustre: server umount lustre-MDT0000 complete [ 2316.719771] LustreError: 64633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623604 with bad export cookie 8885108220265591364 [ 2316.720846] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.727273] LustreError: 64633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2316.772513] LustreError: 71096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.788191] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.791385] LustreError: 71096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2317.052316] Lustre: server umount lustre-MDT0001 complete [ 2323.347877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2327.142778] Lustre: DEBUG MARKER: files compared the same [ 2329.368979] Lustre: server umount lustre-OST0000 complete [ 2346.723869] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2350.626987] Key type lgssc unregistered [ 2350.967702] LNet: 74371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2350.974586] LNetError: 74371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.992742] LNet: Removed LNI 192.168.204.129@tcp [ 2352.127637] Key type .llcrypt unregistered [ 2352.129367] Key type ._llcrypt unregistered [ 2379.870174] Key type ._llcrypt registered [ 2379.874246] Key type .llcrypt registered [ 2380.007702] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2380.971041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2381.145297] alg: No test for adler32 (adler32-zlib) [ 2382.188829] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 2382.522570] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2384.287560] Key type lgssc registered [ 2385.931412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2394.536473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2394.586469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.181642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.634789] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug +malloc all [ 2405.349468] LustreError: 75784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.384635] LustreError: 75784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2410.464930] LustreError: 75783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.593419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.877561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2416.975751] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug +malloc all [ 2425.663045] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2431.856352] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2439.268271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2439.572214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.620677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2446.710433] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug +malloc all [ 2455.048478] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2467.298458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.328218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.418643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.421645] Lustre: Skipped 2 previous similar messages [ 2472.772617] Lustre: server umount lustre-OST0000 complete [ 2477.536329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.539632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.546526] Lustre: Skipped 2 previous similar messages [ 2482.659965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.666960] Lustre: Skipped 2 previous similar messages [ 2483.875381] Lustre: server umount lustre-MDT0000 complete [ 2487.776938] LustreError: 75778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.795990] LustreError: 75778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2488.702886] LustreError: 75765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623776 with bad export cookie 15442688927298961933 [ 2488.713581] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2488.722037] LustreError: 75765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2489.103401] Lustre: server umount lustre-MDT0001 complete [ 2502.993924] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2507.595950] Key type lgssc unregistered [ 2508.041885] LNet: 79316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2508.057457] LNetError: 79316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2508.079291] LNet: Removed LNI 192.168.204.129@tcp [ 2509.198165] Key type .llcrypt unregistered [ 2509.203106] Key type ._llcrypt unregistered [ 2539.584870] Key type ._llcrypt registered [ 2539.594695] Key type .llcrypt registered [ 2539.730607] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2540.688550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2540.733250] alg: No test for adler32 (adler32-zlib) [ 2541.953901] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 2542.229258] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2543.975202] Key type lgssc registered [ 2545.226185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.859068] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2554.910480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.591838] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2575.711354] LustreError: 80704:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 2602.463242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.961564] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2616.712601] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2625.114342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.297497] Lustre: *** cfs_fail_loc=706, val=0*** [ 2625.655780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.740681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2630.420633] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2631.138535] LustreError: 80730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.157700] LustreError: 80730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2636.256606] LustreError: 80729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.287036] LustreError: 80730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.345489] LustreError: 80730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2641.225525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.800109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2646.956434] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2656.323108] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2662.832782] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2666.464769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2666.481993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.503269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2667.490960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2667.503695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2667.550304] Lustre: Skipped 1 previous similar message [ 2671.717364] Lustre: server umount lustre-OST0000 complete [ 2677.728524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.732863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2677.739273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.776161] Lustre: Skipped 1 previous similar message [ 2677.824174] Lustre: Skipped 1 previous similar message [ 2682.870923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.887167] Lustre: Skipped 1 previous similar message [ 2683.037177] Lustre: server umount lustre-MDT0000 complete [ 2687.862929] LustreError: 80710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623975 with bad export cookie 3945894492000292378 [ 2687.875642] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2688.240358] Lustre: server umount lustre-MDT0001 complete [ 2700.854770] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2704.803268] Key type lgssc unregistered [ 2705.240450] LNet: 84321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2705.269467] LNetError: 84321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2705.288409] LNet: Removed LNI 192.168.204.129@tcp [ 2706.421101] Key type .llcrypt unregistered [ 2706.423631] Key type ._llcrypt unregistered [ 2733.938715] Key type ._llcrypt registered [ 2733.941596] Key type .llcrypt registered [ 2734.065652] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2735.608843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2735.677806] alg: No test for adler32 (adler32-zlib) [ 2736.787684] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 2737.127435] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2738.841524] Key type lgssc registered [ 2740.377756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2749.113198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2749.175828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.292646] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2764.515730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.183239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2770.405956] LustreError: 86422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.430077] LustreError: 86422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2770.759630] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2774.496843] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.513797] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2779.617819] LustreError: 86426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.634041] LustreError: 86426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2780.719061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2781.103099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.278569] LustreError: 86426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.301812] LustreError: 86426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2788.087503] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2794.477140] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2794.505828] LustreError: 86425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2797.779935] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2797.794848] Lustre: MGS: Client 797a5083-64ba-409e-9963-15be1ceeeba3 (at 0@lo) reconnecting [ 2797.825744] LustreError: 86436:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2798.065726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.147894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2803.406184] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2819.553341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2819.566162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.581195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2823.658614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.658896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2823.660254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2825.198631] Lustre: server umount lustre-OST0000 complete [ 2833.892255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2833.910387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2833.913627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2833.917930] Lustre: Skipped 2 previous similar messages [ 2839.012613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2839.030345] Lustre: Skipped 1 previous similar message [ 2844.127357] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2844.134373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.151862] Lustre: Skipped 1 previous similar message [ 2844.404837] Lustre: server umount lustre-MDT0000 complete [ 2848.831820] Lustre: server umount lustre-MDT0001 complete [ 2853.170302] Lustre: server umount MGS complete [ 2867.636892] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:02:32 (1788624152) [ 2904.063327] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 2914.755398] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2957.809575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.247634] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2974.748960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.187643] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2975.249223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2975.454954] Lustre: lustre-MDT0001: new disk, initializing [ 2975.548369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2975.572864] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2975.583361] Lustre: 93630:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2975.588668] LustreError: 93630:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2980.961370] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2993.668536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2994.049990] Lustre: lustre-OST0000: new disk, initializing [ 2994.062280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2994.071067] Lustre: 94396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2994.187027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.910413] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3003.943492] Lustre: 94847:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009ec694dd x1875508224633856/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788624307 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3011.276343] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3020.479552] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3020.521892] Lustre: MGS: Client bdd16a7a-378e-428f-b9f8-a6fcc58deafd (at 0@lo) reconnecting [ 3020.552056] LustreError: 93616:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3020.600146] Lustre: 92895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3020.671048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3020.813339] Lustre: lustre-MDT0000: new disk, initializing [ 3021.010247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3021.035465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3021.056398] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3022.115222] Lustre: 95464:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000556063a6 x1875508224640768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788624325 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3026.413362] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3027.191544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3027.203954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3027.297693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3053.038272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3053.047929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.067841] Lustre: Skipped 1 previous similar message [ 3053.071971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3057.138711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3057.155256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.335640] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3066.537765] Lustre: server umount lustre-OST0000 complete [ 3072.491524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3072.496049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3072.498950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3072.498963] Lustre: Skipped 4 previous similar messages [ 3072.502251] Lustre: Skipped 1 previous similar message [ 3078.568107] Lustre: server umount lustre-MDT0000 complete [ 3088.930450] Lustre: server umount lustre-MDT0001 complete [ 3092.727988] Lustre: server umount MGS complete [ 3105.505817] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:06:30 (1788624390) [ 3124.987529] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3127.940884] Key type lgssc unregistered [ 3128.312764] LNet: 97963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3128.328989] LNetError: 97963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3128.356936] LNet: Removed LNI 192.168.204.129@tcp [ 3129.202191] Key type .llcrypt unregistered [ 3129.207132] Key type ._llcrypt unregistered [ 3147.693671] Key type ._llcrypt registered [ 3147.695918] Key type .llcrypt registered [ 3147.851271] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3149.226586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3149.353100] alg: No test for adler32 (adler32-zlib) [ 3150.478953] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 3150.812558] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3152.528639] Key type lgssc registered [ 3153.793718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3162.160842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3162.193811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.888152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.248193] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3173.346995] LustreError: 99183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.367806] LustreError: 99183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3176.949982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.544931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3182.308602] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3191.292414] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3197.263501] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3205.060935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.599593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.699471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3212.870469] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3222.270597] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3233.253565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3233.265110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3233.282546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3233.304519] Lustre: Skipped 1 previous similar message [ 3237.800312] Lustre: server umount lustre-OST0000 complete [ 3243.492810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3243.499875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3243.514275] Lustre: Skipped 2 previous similar messages [ 3243.528989] Lustre: Skipped 1 previous similar message [ 3248.612085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.625538] Lustre: Skipped 1 previous similar message [ 3253.730992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3253.738873] Lustre: Skipped 1 previous similar message [ 3256.287253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3256.437647] Lustre: server umount lustre-MDT0000 complete [ 3258.848285] LustreError: 99182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.877299] LustreError: 99182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3260.280689] LustreError: 99917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624547 with bad export cookie 6321150132516236412 [ 3260.287700] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3260.288972] LustreError: 99917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3260.572891] Lustre: server umount lustre-MDT0001 complete [ 3271.716125] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3274.909888] Key type lgssc unregistered [ 3275.268979] LNet: 102717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3275.285289] LNetError: 102717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3275.308511] LNet: Removed LNI 192.168.204.129@tcp [ 3276.396988] Key type .llcrypt unregistered [ 3276.400541] Key type ._llcrypt unregistered [ 3294.934367] Key type ._llcrypt registered [ 3294.937186] Key type .llcrypt registered [ 3295.079207] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3296.700081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3296.744722] alg: No test for adler32 (adler32-zlib) [ 3298.032635] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 3298.458621] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3300.167185] Key type lgssc registered [ 3302.211614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3309.553975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3309.560281] LustreError: 103806:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3309.573451] LustreError: 103807:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3309.585711] LustreError: 103806:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3309.594432] LustreError: 103806:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3309.614265] LustreError: 103806:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3309.623625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.223190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.802445] LustreError: 103844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.843168] LustreError: 103844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3320.962279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.447637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3325.791572] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3332.591562] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing lsmod [ 3336.547044] LustreError: 105011:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3336.551402] LustreError: 105012:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3336.579977] LustreError: 105011:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3336.600858] LustreError: 105011:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3336.619314] LustreError: 105011:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3336.638822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3337.276395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.792183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3341.798817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.806922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3342.336494] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.337330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3342.357409] Lustre: Skipped 1 previous similar message [ 3342.379634] Lustre: Skipped 2 previous similar messages [ 3346.472849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3346.492266] Lustre: Skipped 2 previous similar messages [ 3347.307798] Lustre: server umount lustre-MDT0000 complete [ 3351.525762] LustreError: 103845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3351.555475] LustreError: 103845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3352.323446] LustreError: 103820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624640 with bad export cookie 7009721136530188913 [ 3352.332609] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.347850] LustreError: 103820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3352.631321] LustreError: 104274:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000076456baf x1875508813910656/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3352.746650] Lustre: server umount lustre-MDT0001 complete [ 3367.369705] Lustre: server umount lustre-OST0000 complete [ 3377.192446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.042320] LustreError: 106077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.182027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3383.266647] LustreError: 106078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.557640] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3387.363110] LustreError: 106077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.872924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.759283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3401.900153] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3412.101566] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3420.308411] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3427.948362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3428.633188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.118783] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3445.393550] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3448.126454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3458.535818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3458.541250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.551760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3458.587295] Lustre: Skipped 1 previous similar message [ 3463.359723] Lustre: server umount lustre-OST0000 complete [ 3468.772672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.794045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.804505] Lustre: Skipped 2 previous similar messages [ 3473.365356] Lustre: server umount lustre-MDT0000 complete [ 3473.888778] LustreError: 106078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.914942] LustreError: 106078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3477.366292] LustreError: 106058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624765 with bad export cookie 7009721136530189613 [ 3477.371865] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3477.378327] LustreError: 106058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3477.771129] Lustre: server umount lustre-MDT0001 complete [ 3488.482987] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3491.144982] Key type lgssc unregistered [ 3491.487485] LNet: 109609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3491.492107] LNetError: 109609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3491.500757] LNet: Removed LNI 192.168.204.129@tcp [ 3492.309704] Key type .llcrypt unregistered [ 3492.312233] Key type ._llcrypt unregistered [ 3518.329023] Key type ._llcrypt registered [ 3518.332176] Key type .llcrypt registered [ 3518.536976] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3519.722632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3519.802801] alg: No test for adler32 (adler32-zlib) [ 3520.933348] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 3521.202914] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3522.943179] Key type lgssc registered [ 3524.478531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3531.315221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3531.339358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.744952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.518387] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3543.009037] LustreError: 111021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3543.025903] LustreError: 111021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3544.496111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.860689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3549.525457] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3556.997922] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3562.773443] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3570.607415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3571.112754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3577.943096] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3578.303589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3585.318466] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3594.549891] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3599.233514] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3603.957339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3603.979106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.013590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3605.997591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3605.999800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3606.002363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.028415] Lustre: Skipped 1 previous similar message [ 3611.111415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3611.130642] Lustre: Skipped 1 previous similar message [ 3616.225738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3616.235728] Lustre: Skipped 1 previous similar message [ 3618.271302] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3618.480962] Lustre: server umount lustre-OST0000 complete [ 3626.466144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3626.475710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3626.480896] Lustre: Skipped 1 previous similar message [ 3628.345830] Lustre: server umount lustre-MDT0000 complete [ 3631.586760] LustreError: 112465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3631.604922] LustreError: 112465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3632.246483] LustreError: 111002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624919 with bad export cookie 72786687652908040 [ 3632.254943] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3632.260315] LustreError: 111002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3632.668781] Lustre: server umount lustre-MDT0001 complete [ 3643.940538] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3646.973432] Key type lgssc unregistered [ 3647.339149] LNet: 114662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3647.361663] LNetError: 114662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3647.381311] LNet: Removed LNI 192.168.204.129@tcp [ 3648.190735] Key type .llcrypt unregistered [ 3648.195955] Key type ._llcrypt unregistered [ 3664.463186] Key type ._llcrypt registered [ 3664.467349] Key type .llcrypt registered [ 3664.650659] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3665.542213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3665.617866] alg: No test for adler32 (adler32-zlib) [ 3666.612636] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 3666.896594] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3668.576378] Key type lgssc registered [ 3669.894264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3680.551436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3680.591553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.147356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.836060] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3691.501737] LustreError: 115883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.521102] LustreError: 115883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3695.752535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3696.139635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3701.904978] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3710.030297] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3716.316706] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3723.980366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3724.331487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.395683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3731.869578] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3740.162701] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3750.883384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.889407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3750.915739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3754.550035] Lustre: server umount lustre-OST0000 complete [ 3761.121739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.140196] Lustre: Skipped 1 previous similar message [ 3761.144425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3761.146667] Lustre: Skipped 1 previous similar message [ 3766.241764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3766.259596] Lustre: Skipped 2 previous similar messages [ 3770.343422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3773.408222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3773.932454] Lustre: server umount lustre-MDT0000 complete [ 3775.462320] LustreError: 115878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3775.482291] LustreError: 115878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3778.666437] LustreError: 115864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625066 with bad export cookie 8011583510535231028 [ 3778.673635] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3778.681295] LustreError: 115864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3778.890481] Lustre: server umount lustre-MDT0001 complete [ 3788.752418] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3791.125217] Key type lgssc unregistered [ 3791.331175] LNet: 119417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3791.343631] LNetError: 119417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3791.370199] LNet: Removed LNI 192.168.204.129@tcp [ 3792.144398] Key type .llcrypt unregistered [ 3792.147083] Key type ._llcrypt unregistered [ 3812.811962] Key type ._llcrypt registered [ 3812.816267] Key type .llcrypt registered [ 3812.948965] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3814.170028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3814.222077] alg: No test for adler32 (adler32-zlib) [ 3815.347494] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 3815.768235] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3817.527152] Key type lgssc registered [ 3818.570595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3826.547483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3826.581201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3828.186934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.396946] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3837.411954] LustreError: 120811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.429650] LustreError: 120811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3842.212050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.668037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3847.844632] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3856.032977] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3861.392418] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3867.882552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3868.145230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.277942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3874.348887] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3881.919084] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3891.996757] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3895.210820] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3895.219759] Lustre: Skipped 2 previous similar messages [ 3898.429322] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3898.472198] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3899.473940] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3899.479714] Lustre: Skipped 2 previous similar messages [ 3904.569564] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3904.578020] Lustre: Skipped 1 previous similar message [ 3909.424439] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.29@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3909.441685] Lustre: Skipped 2 previous similar messages [ 3913.600201] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.29@tcp 192.168.0.[2,10]@tcp [ 3917.505295] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.29/24@tcp in log lustre-MDT0000 [ 3917.514655] Lustre: Skipped 2 previous similar messages [ 3919.679441] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.29/24@tcp [ 3919.691432] Lustre: Skipped 1 previous similar message [ 3926.000207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3926.008064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.012316] LustreError: Skipped 1 previous similar message [ 3926.043263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3927.540245] Lustre: server umount lustre-OST0000 complete [ 3933.665894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.672220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3933.683794] Lustre: Skipped 1 previous similar message [ 3936.854988] Lustre: server umount lustre-MDT0000 complete [ 3938.784480] LustreError: 121551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.798496] LustreError: 121551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3940.020222] LustreError: 123441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625227 with bad export cookie 17324554347486872398 [ 3940.026951] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.034724] LustreError: 123441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3949.669812] Lustre: server umount lustre-MDT0001 complete [ 3958.188839] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3960.343814] Key type lgssc unregistered [ 3960.588471] LNet: 125659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3960.600287] LNetError: 125659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3960.616599] LNet: Removed LNI 192.168.204.129@tcp [ 3961.316876] Key type .llcrypt unregistered [ 3961.321238] Key type ._llcrypt unregistered [ 3987.199590] Key type ._llcrypt registered [ 3987.201900] Key type .llcrypt registered [ 3987.277855] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3988.153127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3988.177645] alg: No test for adler32 (adler32-zlib) [ 3989.366046] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 3989.609890] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3991.319286] Key type lgssc registered [ 3992.569441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4001.388287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4001.440501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.967246] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4002.995054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4003.001380] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.29/24@tcp [ 4007.997982] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4012.526360] LustreError: 127218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.558408] LustreError: 127218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4017.440854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.639404] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4017.784643] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4017.836089] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.29/24@tcp [ 4017.906198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4022.525319] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4030.276669] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4036.924085] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4044.590773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4044.904841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4048.034617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4050.824381] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4059.026143] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4072.936430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4072.937852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.945961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.963607] Lustre: Skipped 1 previous similar message [ 4077.936986] Lustre: server umount lustre-OST0000 complete [ 4083.170539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.175383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4083.186638] Lustre: Skipped 1 previous similar message [ 4083.204800] Lustre: Skipped 2 previous similar messages [ 4087.623134] Lustre: server umount lustre-MDT0000 complete [ 4088.296533] LustreError: 127217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.315310] LustreError: 127217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4091.400147] LustreError: 127198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625379 with bad export cookie 9206889689968234585 [ 4091.408786] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4091.411110] LustreError: 127198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4091.664737] Lustre: server umount lustre-MDT0001 complete [ 4102.075923] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4104.699235] Key type lgssc unregistered [ 4104.963857] LNet: 130805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4104.982405] LNetError: 130805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4105.002231] LNet: Removed LNI 192.168.204.129@tcp [ 4105.690099] Key type .llcrypt unregistered [ 4105.692223] Key type ._llcrypt unregistered [ 4134.099923] Key type ._llcrypt registered [ 4134.103297] Key type .llcrypt registered [ 4134.195206] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 4150.235658] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4150.991657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4151.066656] alg: No test for adler32 (adler32-zlib) [ 4152.090885] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 4152.349258] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4154.071213] Key type lgssc registered [ 4155.103780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4191.727063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4191.811792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4193.177351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4193.223227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4193.348050] Lustre: lustre-MDT0000: new disk, initializing [ 4193.416026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.439849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4198.296656] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4208.784805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.917384] Lustre: 134211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4209.003371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4209.003835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4209.011979] Lustre: Skipped 1 previous similar message [ 4209.088444] Lustre: lustre-MDT0001: new disk, initializing [ 4209.180409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4209.218431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4209.240817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4213.843565] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4223.260405] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4228.693191] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4234.588741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.825762] Lustre: lustre-OST0000: new disk, initializing [ 4234.829157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4234.835430] Lustre: 136181:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4234.905393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.616780] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4243.544149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4243.556762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4243.674989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4249.527638] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4254.051581] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4254.471241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4260.387842] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4260.760419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4264.417820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4264.435025] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.447658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4264.463607] Lustre: Skipped 1 previous similar message [ 4264.481145] Lustre: Skipped 1 previous similar message [ 4269.547083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.557088] Lustre: Skipped 1 previous similar message [ 4274.661393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.685943] Lustre: Skipped 1 previous similar message [ 4277.215140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4277.478581] Lustre: server umount lustre-OST0000 complete [ 4284.896955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.922663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4284.927935] Lustre: Skipped 1 previous similar message [ 4288.495498] Lustre: server umount lustre-MDT0000 complete [ 4292.184475] LustreError: 134203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625579 with bad export cookie 2552837619681256256 [ 4292.188390] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4292.198000] LustreError: 134203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4292.576489] Lustre: server umount lustre-MDT0001 complete [ 4300.756654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4301.165331] LustreError: 138401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.184368] LustreError: 138401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4301.228756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.465651] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4306.401092] LustreError: 138402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4310.496875] LustreError: 138401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.644138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4313.987638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4318.472257] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4324.764566] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4331.437987] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4340.517521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4340.848464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.922307] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4355.458693] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4359.415259] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4359.830708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4365.081695] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4365.348597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4376.219389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4376.548810] Lustre: lustre-OST0001: new disk, initializing [ 4376.575713] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4376.589903] Lustre: 141772:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4376.741802] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4383.664352] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4387.001495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4387.019379] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4387.148783] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4392.886367] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4393.063704] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4397.110078] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4397.360294] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4402.774823] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99c242d21000.ost_server_uuid 50 [ 4404.171266] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99c242d21000.ost_server_uuid in FULL state after 0 sec [ 4418.017101] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4418.026927] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.034497] LustreError: Skipped 2 previous similar messages [ 4418.035138] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4418.073209] Lustre: Skipped 2 previous similar messages [ 4419.840310] Lustre: server umount lustre-OST0001 complete [ 4423.143016] LustreError: 143075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.158400] LustreError: 143075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4426.213193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4426.232886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.259924] LustreError: 140221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.286642] LustreError: 140221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4430.674769] Lustre: server umount lustre-OST0000 complete [ 4436.967815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4436.980074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.003239] Lustre: Skipped 1 previous similar message [ 4437.008944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4437.011056] Lustre: Skipped 4 previous similar messages [ 4441.705576] Lustre: server umount lustre-MDT0000 complete [ 4443.616474] LustreError: 138402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.646163] LustreError: 138402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4445.897633] LustreError: 141771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625733 with bad export cookie 2552837619681257138 [ 4445.918056] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4445.923759] LustreError: 141771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4446.259445] Lustre: server umount lustre-MDT0001 complete [ 4464.599728] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4467.912026] Key type lgssc unregistered [ 4468.241936] LNet: 144773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4468.250677] LNetError: 144773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4468.276632] LNet: Removed LNI 192.168.204.129@tcp [ 4469.265139] Key type .llcrypt unregistered [ 4469.271859] Key type ._llcrypt unregistered [ 4515.965097] Key type ._llcrypt registered [ 4515.968024] Key type .llcrypt registered [ 4516.081664] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 4530.044490] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4530.807829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4530.820532] alg: No test for adler32 (adler32-zlib) [ 4531.856386] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 4532.079827] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4533.727327] Key type lgssc registered [ 4534.768587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4565.723783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4565.740460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.903834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4566.930231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4567.012983] Lustre: lustre-MDT0000: new disk, initializing [ 4567.089082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.102227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4570.906348] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4579.931718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4580.019726] Lustre: 148926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4580.069864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4580.074187] Lustre: Skipped 1 previous similar message [ 4580.138319] Lustre: lustre-MDT0001: new disk, initializing [ 4580.219449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4580.270934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4580.285263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4584.074911] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4592.934804] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4598.517321] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4604.743622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4604.955896] Lustre: lustre-OST0000: new disk, initializing [ 4604.959842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4604.966757] Lustre: 150894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4605.053866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4610.953694] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4612.713531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4612.732800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4612.830261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4620.607875] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4625.169023] Lustre: Failing over lustre-OST0000 [ 4626.926093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4626.938305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.959623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4627.416705] Lustre: server umount lustre-OST0000 complete [ 4644.831120] Lustre: 147149:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788625916/real 1788625916] req@00000000e9772607 x1875510106817152/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788625932 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4644.845843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.314768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4646.702270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4646.716971] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4649.759692] Lustre: 147149:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788625921/real 1788625921] req@00000000e3c157c3 x1875510106818816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788625937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4652.491194] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4654.101174] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4654.559163] Lustre: 147148:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788625926/real 1788625926] req@00000000165aeb72 x1875510106819712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788625942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4656.129782] Lustre: Failing over lustre-MDT0000 [ 4656.366720] Lustre: server umount lustre-MDT0000 complete [ 4657.121232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4657.129725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.999614] Lustre: 147148:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788625931/real 1788625931] req@00000000f8ae49bb x1875510106820608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788625947 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4660.196935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4660.218440] Lustre: Skipped 1 previous similar message [ 4664.337828] LustreError: 148932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.360308] LustreError: 148932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4665.314527] LustreError: 148937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.340405] LustreError: 148937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4669.452213] LustreError: 148933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.508831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4674.575730] LustreError: 150295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.588163] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4674.592246] LustreError: 150295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4674.756394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4674.804477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4674.868706] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4679.328438] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4679.693995] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4680.177386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4680.214403] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4680.281322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4690.402290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4690.422038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.435549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4690.444638] Lustre: Skipped 1 previous similar message [ 4695.524285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4695.534047] Lustre: Skipped 1 previous similar message [ 4700.645942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4700.655258] Lustre: Skipped 1 previous similar message [ 4701.151278] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4701.366653] Lustre: server umount lustre-OST0000 complete [ 4705.763338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4705.769632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.772709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4705.773528] LustreError: Skipped 1 previous similar message [ 4705.807926] Lustre: Skipped 2 previous similar messages [ 4711.249072] Lustre: server umount lustre-MDT0000 complete [ 4715.139552] LustreError: 148916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626002 with bad export cookie 17031176013650098422 [ 4715.145906] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4715.170213] LustreError: 148916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4715.542817] Lustre: server umount lustre-MDT0001 complete [ 4725.800733] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4728.706053] Key type lgssc unregistered [ 4729.011030] LNet: 154550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4729.022148] LNetError: 154550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4730.102786] LNet: Removed LNI 192.168.204.129@tcp [ 4731.510099] Key type .llcrypt unregistered [ 4731.515955] Key type ._llcrypt unregistered [ 4772.008496] Key type ._llcrypt registered [ 4772.011253] Key type .llcrypt registered [ 4772.154102] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4773.169919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4773.416198] alg: No test for adler32 (adler32-zlib) [ 4774.530806] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 4774.769373] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4776.463775] Key type lgssc registered [ 4777.618718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4786.018886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4786.048556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.536387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.643933] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4797.922990] LustreError: 156069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.947563] LustreError: 156069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4801.499300] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4803.040629] LustreError: 156070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4805.728124] LustreError: 156098:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4805.740319] LustreError: 156098:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4805.749806] LustreError: 156098:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4806.024464] Lustre: server umount lustre-MDT0000 complete [ 4821.351625] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 4832.602612] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4873.247577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4873.448531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4873.497113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4873.585136] Lustre: lustre-MDT0000: new disk, initializing [ 4873.718845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4873.743525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4878.230541] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4887.953702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.068429] Lustre: 159403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4888.109170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4888.113078] Lustre: Skipped 1 previous similar message [ 4888.218039] Lustre: lustre-MDT0001: new disk, initializing [ 4888.311407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4888.361051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4888.387286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4892.860214] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4903.239642] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4910.811839] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4919.143747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4919.422723] Lustre: lustre-OST0000: new disk, initializing [ 4919.427702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4919.440884] Lustre: 161371:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4919.545521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4927.168644] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4929.624785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4929.645751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4929.884989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4938.126321] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4942.609443] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4944.406770] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:37:09 (1788626229) [ 5200.863675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5200.874091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.893335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5201.376609] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.396103] Lustre: Skipped 1 previous similar message [ 5201.398153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5201.414923] Lustre: Skipped 2 previous similar messages [ 5204.727620] Lustre: server umount lustre-MDT0000 complete [ 5206.495898] LustreError: 159414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5206.512911] LustreError: 159414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5207.451033] LustreError: 159395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626495 with bad export cookie 2581914456654748280 [ 5207.453637] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5207.459441] LustreError: 159395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5211.619025] LustreError: 163136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5211.620806] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.659892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5213.745689] Lustre: server umount lustre-MDT0001 complete [ 5231.071173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5231.247756] Lustre: server umount lustre-OST0000 complete [ 5240.503599] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 5250.001735] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5284.249850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.528686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5284.553733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5284.622584] Lustre: lustre-MDT0000: new disk, initializing [ 5284.717754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5284.759511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5289.735719] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5301.255361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.410652] Lustre: 166038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5301.501411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5301.515562] Lustre: Skipped 1 previous similar message [ 5301.619508] Lustre: lustre-MDT0001: new disk, initializing [ 5301.731500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5301.779063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5301.788260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5306.402662] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5315.616374] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5321.407613] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5328.223442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5328.600484] Lustre: lustre-OST0000: new disk, initializing [ 5328.603862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5328.613876] Lustre: 168005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5328.683452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5334.880680] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5338.683923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5338.696552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5338.781190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5343.647261] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5348.441163] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5348.698878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5353.323613] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5353.495171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5359.589490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.591434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5359.596801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5359.596810] Lustre: Skipped 1 previous similar message [ 5359.614531] Lustre: Skipped 1 previous similar message [ 5359.637052] LustreError: Skipped 1 previous similar message [ 5361.546912] Lustre: server umount lustre-OST0000 complete [ 5368.289709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5368.296466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.305768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5368.309791] Lustre: Skipped 1 previous similar message [ 5371.384828] Lustre: server umount lustre-MDT0000 complete [ 5374.945253] LustreError: 166848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.957912] LustreError: 166848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5375.086342] LustreError: 168004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626662 with bad export cookie 2581914456654907110 [ 5375.087283] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.097921] LustreError: 168004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5375.471721] Lustre: server umount lustre-MDT0001 complete [ 5384.253399] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:44:29 (1788626669) [ 5399.820842] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 5410.594983] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5446.110247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5446.428507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5446.458710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5446.545762] Lustre: lustre-MDT0000: new disk, initializing [ 5446.633559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5446.647670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5451.394222] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5463.080192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.228216] Lustre: 172766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5463.498810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5467.419530] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5477.703730] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5483.141342] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5489.528355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5489.818052] Lustre: lustre-OST0000: new disk, initializing [ 5489.823760] Lustre: Skipped 1 previous similar message [ 5489.833506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5489.845342] Lustre: Skipped 2 previous similar messages [ 5489.851551] Lustre: 174742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5495.581375] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5498.494434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5498.505471] Lustre: Skipped 1 previous similar message [ 5498.515518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5498.704211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5505.588776] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5514.210758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5514.230180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5514.253775] Lustre: Skipped 1 previous similar message [ 5514.263351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5514.273626] Lustre: Skipped 2 previous similar messages [ 5519.383214] Lustre: server umount lustre-OST0000 complete [ 5534.694134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5534.703483] Lustre: Skipped 6 previous similar messages [ 5538.272031] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5538.573320] Lustre: server umount lustre-MDT0000 complete [ 5539.808711] LustreError: 172777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.828464] LustreError: 172777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5541.873367] LustreError: 172756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626829 with bad export cookie 2581914456654907985 [ 5541.881222] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5541.884694] LustreError: 172756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5542.148689] Lustre: server umount lustre-MDT0001 complete [ 5550.306232] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:47:15 (1788626835) [ 5567.583805] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 5583.044665] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5622.822885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.099756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5623.145538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5623.240960] Lustre: lustre-MDT0000: new disk, initializing [ 5623.324346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.329106] Lustre: Skipped 2 previous similar messages [ 5623.355726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5628.138426] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5639.081829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.322155] Lustre: 179086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5639.564625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5645.640489] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5655.742427] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5661.532326] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5668.857224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5669.168499] Lustre: 181067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5674.751097] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5677.620953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5677.710783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5683.074700] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5692.905571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5692.906567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5692.919353] Lustre: Skipped 4 previous similar messages [ 5692.928024] Lustre: Skipped 1 previous similar message [ 5696.466578] Lustre: server umount lustre-OST0000 complete [ 5701.088590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5701.096586] LustreError: Skipped 1 previous similar message [ 5705.712366] Lustre: server umount lustre-MDT0000 complete [ 5708.259266] LustreError: 179100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5708.274843] LustreError: 179100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5708.663605] LustreError: 179081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626996 with bad export cookie 2581914456654909077 [ 5708.665713] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5708.679830] LustreError: 179081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5708.976531] Lustre: server umount lustre-MDT0001 complete [ 5718.788682] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5721.467229] Key type lgssc unregistered [ 5721.831541] LNet: 182920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5721.840898] LNetError: 182920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5721.862632] LNet: Removed LNI 192.168.204.129@tcp [ 5722.690617] Key type .llcrypt unregistered [ 5722.693805] Key type ._llcrypt unregistered [ 5744.509772] Key type ._llcrypt registered [ 5744.512596] Key type .llcrypt registered [ 5744.602739] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5745.483214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5745.505149] alg: No test for adler32 (adler32-zlib) [ 5746.529249] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 5746.715549] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5748.383167] Key type lgssc registered [ 5749.472816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5756.475383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5756.485754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.842418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5762.330732] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5768.160850] LustreError: 184332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.180072] LustreError: 184332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5770.533774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5770.933606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5775.785500] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5782.855456] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5788.486672] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5794.721993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5794.998185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5800.640798] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5803.209551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5808.189408] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5827.042102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5827.050304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.068982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5829.099296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5829.099381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5829.110080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5831.749326] Lustre: server umount lustre-OST0000 complete [ 5837.282811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5837.295886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5837.316899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5837.322033] Lustre: Skipped 1 previous similar message [ 5839.331594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5839.334846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.953706] Lustre: server umount lustre-MDT0000 complete [ 5844.449292] LustreError: 185339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5844.484123] LustreError: 185339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5845.725799] LustreError: 184314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627133 with bad export cookie 9191916045705475891 [ 5845.737922] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5845.748405] LustreError: 184314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5846.158685] Lustre: server umount lustre-MDT0001 complete [ 5858.069421] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5860.948837] Key type lgssc unregistered [ 5861.200328] LNet: 187886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5861.206509] LNetError: 187886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5861.222642] LNet: Removed LNI 192.168.204.129@tcp [ 5861.964431] Key type .llcrypt unregistered [ 5861.967142] Key type ._llcrypt unregistered [ 5883.485663] Key type ._llcrypt registered [ 5883.488374] Key type .llcrypt registered [ 5883.589750] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5884.399793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5884.506496] alg: No test for adler32 (adler32-zlib) [ 5885.594637] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 5885.874433] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5887.585238] Key type lgssc registered [ 5888.924905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5897.444938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5897.464970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.941726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.618165] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5908.451659] LustreError: 189282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.470857] LustreError: 189282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5913.582948] LustreError: 189281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.086444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.419748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5919.302073] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5925.864466] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5931.467561] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5938.369340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5938.885748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5944.028442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5945.741620] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5953.669445] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5960.673908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5960.677456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.687869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.776754] Lustre: lustre-OST0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 5962.730089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.708179] Lustre: server umount lustre-OST0000 complete [ 5974.587211] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99c244174800.ost_server_uuid 50 [ 5976.224166] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5991.394345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5991.402789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.415634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5991.426083] Lustre: Skipped 2 previous similar messages [ 5995.780103] Lustre: server umount lustre-MDT0000 complete [ 5997.536331] LustreError: 189282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.551706] LustreError: 189282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5999.160241] LustreError: 189263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627286 with bad export cookie 5271967951667039581 [ 5999.161995] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5999.168950] LustreError: 189263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5999.470674] Lustre: server umount lustre-MDT0001 complete [ 6007.498072] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:54:53 (1788627293) [ 6013.585558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.910922] LustreError: 193195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.968333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6017.820833] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6019.043688] LustreError: 193196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.031675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.291135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6029.057434] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6035.685348] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6040.368244] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6046.184816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6046.536407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6052.575381] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6058.818580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6060.068233] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6066.242060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6066.431297] Lustre: lustre-OST0001: new disk, initializing [ 6066.434368] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6066.439433] Lustre: 196034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6066.520618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6072.161559] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6075.531047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6075.539985] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6075.657667] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6080.848804] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6084.584236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6084.589210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.601205] Lustre: Skipped 1 previous similar message [ 6084.606196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6084.614276] Lustre: Skipped 2 previous similar messages [ 6089.993613] Lustre: server umount lustre-OST0000 complete [ 6091.241776] LustreError: 195017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.264970] LustreError: 195017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6096.408536] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6096.651102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6100.828239] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6101.067440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6107.665351] LustreError: 195015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6107.695484] LustreError: 195015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6116.846665] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.847445] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6116.857479] Lustre: Skipped 1 previous similar message [ 6116.864328] LustreError: Skipped 1 previous similar message [ 6116.865646] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6116.876301] Lustre: Skipped 5 previous similar messages [ 6120.327889] Lustre: server umount lustre-OST0001 complete [ 6127.074235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.075595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6127.097439] Lustre: Skipped 1 previous similar message [ 6127.114277] Lustre: Skipped 1 previous similar message [ 6129.664537] Lustre: server umount lustre-MDT0000 complete [ 6133.535855] LustreError: 195037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627421 with bad export cookie 5271967951667040652 [ 6133.546118] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6133.550000] LustreError: 195037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6133.853446] Lustre: server umount lustre-MDT0001 complete [ 6158.894230] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:57:23 (1788627443) [ 6165.593712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6165.894335] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6165.928960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6166.104563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.864365] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6180.481526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.606157] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6180.646696] Lustre: 199474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6184.920757] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6192.535825] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6198.612933] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6205.670354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6205.860415] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6211.780566] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6216.315032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6218.884579] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6225.765886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6225.868072] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6232.825523] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6236.382751] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6239.536664] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6242.275705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6242.284822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.298763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6248.054791] Lustre: server umount lustre-OST0000 complete [ 6250.515065] LustreError: 201326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6250.525546] LustreError: 201326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6268.839975] Lustre: server umount lustre-OST0001 complete [ 6272.995170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6275.554162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.559876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6275.566060] Lustre: Skipped 4 previous similar messages [ 6275.579122] Lustre: Skipped 7 previous similar messages [ 6286.303125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6286.438984] Lustre: server umount lustre-MDT0000 complete [ 6290.118946] LustreError: 199465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627577 with bad export cookie 5271967951667042052 [ 6290.120585] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6290.131908] LustreError: 199465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6290.622806] Lustre: server umount lustre-MDT0001 complete [ 6313.126583] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:59:58 (1788627598) [ 6327.223162] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 6335.351109] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6362.039877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.185570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6362.217817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6362.292964] Lustre: lustre-MDT0000: new disk, initializing [ 6362.358963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6362.367982] Lustre: Skipped 3 previous similar messages [ 6362.381558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6365.523431] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6372.946180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.026892] Lustre: 207567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6373.076232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6373.081784] Lustre: Skipped 1 previous similar message [ 6373.135938] Lustre: lustre-MDT0001: new disk, initializing [ 6373.230271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6373.239396] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6376.942589] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6384.932417] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6389.519889] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6395.173414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6395.547645] Lustre: lustre-OST0000: new disk, initializing [ 6395.556532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6395.568190] Lustre: 209548:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6400.692668] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6401.080345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6401.097178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6401.184585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6408.867589] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6412.199805] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6412.332941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6415.945618] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6416.125314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6421.986864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6421.997404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6422.003174] LustreError: Skipped 1 previous similar message [ 6422.005330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6422.015352] Lustre: Skipped 1 previous similar message [ 6422.020206] Lustre: Skipped 4 previous similar messages [ 6423.925838] Lustre: server umount lustre-OST0000 complete [ 6433.555945] Lustre: server umount lustre-MDT0000 complete [ 6436.904389] LustreError: 208363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627724 with bad export cookie 5271967951667043837 [ 6436.905514] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6436.910481] LustreError: 208363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6437.152187] Lustre: server umount lustre-MDT0001 complete [ 6444.337058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.705402] LustreError: 211763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.733960] LustreError: 211763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6448.426581] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6456.094989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6461.118691] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6467.434122] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6472.361136] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6478.455897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6484.060888] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6490.214407] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6493.991255] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6494.207897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6498.500863] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6498.713344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6502.368386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6502.381906] LustreError: Skipped 1 previous similar message [ 6506.759250] Lustre: server umount lustre-OST0000 complete [ 6512.979173] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6517.533733] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6546.925736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6547.226894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.234460] Lustre: Skipped 5 previous similar messages [ 6553.123618] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6555.428793] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.450404] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6559.991843] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6564.502243] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6570.851278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.856183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6570.884389] Lustre: Skipped 4 previous similar messages [ 6570.888654] Lustre: Skipped 6 previous similar messages [ 6582.239295] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6582.406333] Lustre: server umount lustre-OST0000 complete [ 6589.924937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6589.943409] LustreError: Skipped 1 previous similar message [ 6592.729667] Lustre: server umount lustre-MDT0000 complete [ 6596.577049] LustreError: 211764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6596.603272] LustreError: 211764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6596.691335] LustreError: 211743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627884 with bad export cookie 5271967951667044719 [ 6596.694542] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6596.698474] LustreError: 211743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6597.160637] Lustre: server umount lustre-MDT0001 complete [ 6607.208271] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 13:04:52 (1788627892) [ 6614.402395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.653310] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6628.168271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.317729] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6639.518924] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6645.163203] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6651.890774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6658.819662] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6667.285984] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6672.120861] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6672.362511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6677.220303] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6677.457574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6685.303497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6685.475579] Lustre: lustre-OST0001: new disk, initializing [ 6685.478872] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6685.483771] Lustre: 221104:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6691.948089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6691.969925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6692.103560] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6692.576127] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6702.959455] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6709.056731] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6709.397164] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6714.552144] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6714.775940] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6717.932639] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6717.942072] LustreError: Skipped 1 previous similar message [ 6722.751123] Lustre: server umount lustre-OST0001 complete [ 6730.105274] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6730.378109] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6735.695719] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6735.990545] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6766.039075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6771.703382] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6771.717593] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6771.722575] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6771.739761] Lustre: Skipped 1 previous similar message [ 6773.322394] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6780.898734] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6782.446978] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6790.353427] Lustre: server umount lustre-OST0001 complete [ 6813.977061] Lustre: server umount lustre-OST0000 complete [ 6828.000873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6828.019460] Lustre: Skipped 21 previous similar messages [ 6832.095176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6832.369547] Lustre: server umount lustre-MDT0000 complete [ 6836.249037] LustreError: 217713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628123 with bad export cookie 5271967951667045685 [ 6836.253962] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6836.270410] LustreError: 217713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6842.845266] Lustre: server umount lustre-MDT0001 complete [ 6870.836897] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 13:09:16 (1788628156) [ 6879.532907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6879.881745] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6879.942879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6880.254194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6880.261309] Lustre: Skipped 5 previous similar messages [ 6885.126043] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6895.729736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.902534] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6895.945567] Lustre: 226348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6902.469179] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6911.608684] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6918.732699] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6926.183111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.426093] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6932.613896] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6939.739431] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6948.228240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.310486] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6954.248568] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6961.724167] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6963.034881] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6967.316426] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6967.499763] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6972.022213] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6972.252550] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6977.678813] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99c24518b000.ost_server_uuid 50 [ 6979.000529] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99c24518b000.ost_server_uuid in IDLE state after 0 sec [ 6979.950583] Lustre: Permanently deactivating lustre-OST0001 [ 6979.961812] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6983.898692] Lustre: Permanently reactivating lustre-OST0001 [ 6988.783741] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.787613] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6988.813846] Lustre: Skipped 10 previous similar messages [ 6988.820880] LustreError: Skipped 5 previous similar messages [ 6990.663777] Lustre: server umount lustre-OST0001 complete [ 6993.893331] LustreError: 228196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.920752] LustreError: 228196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7009.247637] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7009.397455] Lustre: server umount lustre-OST0000 complete [ 7018.601926] Lustre: server umount lustre-MDT0000 complete [ 7022.094424] LustreError: 226342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628309 with bad export cookie 5271967951667046973 [ 7022.094884] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7022.117844] LustreError: 226342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7022.460559] Lustre: server umount lustre-MDT0001 complete [ 7046.999316] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 13:12:12 (1788628332) [ 7054.754378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7055.031469] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7055.061373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7055.075207] Lustre: Skipped 5 previous similar messages [ 7060.509267] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7069.532357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.640622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7069.682891] Lustre: 233043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7074.744726] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7082.204223] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7087.826969] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7095.271894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7095.574190] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7095.606295] Lustre: Permanently deactivating lustre-OST0000 [ 7095.633192] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7102.159394] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7104.075185] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7109.920651] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7117.340564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7117.420392] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7124.211753] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7124.367292] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7133.072613] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.291055] Lustre: Permanently reactivating lustre-OST0000 [ 7135.318589] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7135.324969] Lustre: Skipped 2 previous similar messages [ 7145.064640] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7145.073903] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 00000000de5ad214, cur 1788628432 deadline 1788628421 last 1788628391 [ 7145.136993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7152.517928] Lustre: server umount lustre-OST0001 complete [ 7162.876450] Lustre: server umount lustre-OST0000 complete [ 7173.398601] Lustre: server umount lustre-MDT0000 complete [ 7177.329346] LustreError: 233038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628465 with bad export cookie 5271967951667048646 [ 7177.337568] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7177.641682] Lustre: server umount lustre-MDT0001 complete [ 7188.618659] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7191.470399] Key type lgssc unregistered [ 7191.761359] LNet: 237923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7191.767417] LNetError: 237923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7191.798061] LNet: Removed LNI 192.168.204.129@tcp [ 7192.758625] Key type .llcrypt unregistered [ 7192.760164] Key type ._llcrypt unregistered [ 7217.261145] Key type ._llcrypt registered [ 7217.266802] Key type .llcrypt registered [ 7217.415154] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7218.758076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7218.788245] alg: No test for adler32 (adler32-zlib) [ 7219.930416] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 7220.155089] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7221.879201] Key type lgssc registered [ 7223.218544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7234.085637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7234.132269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.533286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7240.068778] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7245.797696] LustreError: 239385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.817523] LustreError: 239385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7249.683458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.843506] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7249.864578] Lustre: 239374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7249.873368] Lustre: Permanently deactivating lustre-MDT0001 [ 7249.883905] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7250.141188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7250.972229] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7250.987609] LustreError: 239415:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7250.999732] LustreError: 239415:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7251.021754] LustreError: 239415:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7254.569648] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7261.938812] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7268.188752] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7274.933275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7275.220605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7280.354884] LustreError: 241228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.374159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7282.122609] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7285.733545] LustreError: 241493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.175489] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7290.857762] LustreError: 241227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.958294] LustreError: 241228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.978246] LustreError: 241228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7296.480582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7296.899468] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7302.151528] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7302.157456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7303.887949] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7311.470388] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7315.461033] Lustre: Permanently reactivating lustre-MDT0001 [ 7315.483690] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7315.490249] Lustre: Skipped 2 previous similar messages [ 7317.381741] Lustre: lustre-MDT0001: Client 098ccc7d-685b-4152-baf2-66a5baa7ea09 (at 192.168.204.29@tcp) reconnecting [ 7322.973681] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7324.332895] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7329.566372] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7329.749291] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7334.249673] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7334.417869] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7337.266684] Lustre: Permanently deactivating lustre-MDT0001 [ 7337.307280] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7337.317490] Lustre: Skipped 2 previous similar messages [ 7347.334912] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7357.414777] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.417257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7357.426158] Lustre: Skipped 1 previous similar message [ 7357.444129] Lustre: Skipped 3 previous similar messages [ 7361.113308] Lustre: server umount lustre-MDT0000 complete [ 7362.536955] LustreError: 239379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.552862] LustreError: 239379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7365.786773] LustreError: 244071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628653 with bad export cookie 12539972365379527063 [ 7365.788316] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7365.807739] LustreError: 244071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7366.434470] Lustre: server umount lustre-MDT0001 complete [ 7373.796300] Lustre: server umount lustre-OST0000 complete [ 7387.592421] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 13:17:52 (1788628672) [ 7398.603787] Lustre: server umount lustre-OST0001 complete [ 7405.988443] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 7414.672971] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7450.726548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.985360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7450.993395] Lustre: Skipped 2 previous similar messages [ 7451.027795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7451.124513] Lustre: lustre-MDT0000: new disk, initializing [ 7451.207940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7451.219812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7455.531493] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7465.889810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.084831] Lustre: 247689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7466.206162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7466.211516] Lustre: Skipped 1 previous similar message [ 7466.325828] Lustre: lustre-MDT0001: new disk, initializing [ 7466.493619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7466.631135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7466.655069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7472.307425] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7482.740337] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7488.662065] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7495.735411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.950739] Lustre: lustre-OST0000: new disk, initializing [ 7495.954538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7495.959445] Lustre: 249661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7496.028952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.065409] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7503.442112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7503.451740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7503.566595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7515.242792] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7518.151497] LustreError: 247697:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7528.248401] LustreError: 247697:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7528.320496] LustreError: 250588:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7528.775931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7529.061030] Lustre: lustre-OST0001: new disk, initializing [ 7529.068077] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7529.086282] Lustre: 250777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7529.249223] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7535.253840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7535.269621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7535.401642] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7537.873724] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7538.391385] LustreError: 250588:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7551.896051] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7553.814269] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7556.065344] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7556.079235] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.091707] Lustre: Skipped 2 previous similar messages [ 7556.099310] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7556.106676] Lustre: Skipped 1 previous similar message [ 7561.217580] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7561.227507] Lustre: Skipped 1 previous similar message [ 7562.342583] Lustre: server umount lustre-OST0001 complete [ 7566.308567] LustreError: 249644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.339556] LustreError: 249644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7580.642361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7580.648734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.663167] LustreError: Skipped 2 previous similar messages [ 7580.666472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7580.720883] Lustre: Skipped 2 previous similar messages [ 7586.578426] Lustre: server umount lustre-OST0000 complete [ 7591.905135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.914191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7591.924975] Lustre: Skipped 1 previous similar message [ 7591.928992] Lustre: Skipped 3 previous similar messages [ 7602.161047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7602.186744] Lustre: Skipped 3 previous similar messages [ 7605.215538] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7605.726907] Lustre: server umount lustre-MDT0000 complete [ 7606.322652] LustreError: 247702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7606.351696] LustreError: 247702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7610.924273] LustreError: 247683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628898 with bad export cookie 12539972365379529667 [ 7610.936457] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7610.939738] LustreError: 247683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7617.501654] Lustre: server umount lustre-MDT0001 complete [ 7630.971190] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7635.143327] Key type lgssc unregistered [ 7635.591789] LNet: 252811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7635.599774] LNetError: 252811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7635.612967] LNet: Removed LNI 192.168.204.129@tcp [ 7636.608253] Key type .llcrypt unregistered [ 7636.611462] Key type ._llcrypt unregistered [ 7687.499421] Key type ._llcrypt registered [ 7687.501915] Key type .llcrypt registered [ 7687.656237] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7688.992315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7689.187037] alg: No test for adler32 (adler32-zlib) [ 7690.328046] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 7690.567432] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7692.207190] Key type lgssc registered [ 7693.534564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7703.291995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7703.315403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7704.624249] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7704.640348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7704.837753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7710.000756] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7721.124940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.245545] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7721.288826] Lustre: 255014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7721.632702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7726.981867] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7735.556704] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7742.974928] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7750.602984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7750.881979] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7751.160185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7757.948541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7758.278966] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7768.079558] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7824.365268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7824.375842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7824.398293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7824.866865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7824.878069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.555876] Lustre: server umount lustre-OST0000 complete [ 7831.680339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7847.041710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7847.441587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.012808] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7863.164666] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7864.842556] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7864.867438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7864.870796] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7864.913681] Lustre: Skipped 1 previous similar message [ 7905.766612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7905.769784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7905.789316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7905.789327] Lustre: Skipped 1 previous similar message [ 7905.813520] Lustre: Skipped 1 previous similar message [ 7910.244417] Lustre: server umount lustre-OST0000 complete [ 7916.003440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.011044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.021690] Lustre: Skipped 1 previous similar message [ 7916.032426] Lustre: Skipped 1 previous similar message [ 7919.010406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7919.025584] Lustre: Skipped 1 previous similar message [ 7920.659260] Lustre: server umount lustre-MDT0000 complete [ 7924.776153] LustreError: 255008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629212 with bad export cookie 5808236404612125562 [ 7924.781881] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7924.796626] LustreError: 255008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7925.354739] Lustre: server umount lustre-MDT0001 complete [ 7937.126714] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7940.150501] Key type lgssc unregistered [ 7940.475797] LNet: 260443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7940.485295] LNetError: 260443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.515134] LNet: Removed LNI 192.168.204.129@tcp [ 7941.375408] Key type .llcrypt unregistered [ 7941.377392] Key type ._llcrypt unregistered [ 7967.194559] Key type ._llcrypt registered [ 7967.198233] Key type .llcrypt registered [ 7967.301661] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7968.191587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7968.247499] alg: No test for adler32 (adler32-zlib) [ 7969.294468] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 7969.544771] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7971.200981] Key type lgssc registered [ 7972.470994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7979.582593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7979.606050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7981.046403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7985.982619] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7991.265772] LustreError: 261859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7991.299230] LustreError: 261859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7995.386532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7996.010171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8000.783868] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8009.389390] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8015.395752] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8023.490915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8023.878625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.923262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8031.307334] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8040.489699] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8072.160745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8072.164789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8072.186360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8072.220650] Lustre: Skipped 1 previous similar message [ 8072.231804] Lustre: Skipped 1 previous similar message [ 8073.346676] Lustre: server umount lustre-OST0000 complete [ 8077.798593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8077.812464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.845681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.400252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.404768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8087.521420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8087.537720] Lustre: Skipped 2 previous similar messages [ 8092.127790] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8092.493315] Lustre: server umount lustre-MDT0000 complete [ 8092.649008] LustreError: 264351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8096.967431] LustreError: 261839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629384 with bad export cookie 15981733620488165894 [ 8096.985661] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8097.224173] Lustre: server umount lustre-MDT0001 complete [ 8109.641191] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8112.063305] Key type lgssc unregistered [ 8112.342337] LNet: 266169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8112.349498] LNetError: 266169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8112.378901] LNet: Removed LNI 192.168.204.129@tcp [ 8113.291648] Key type .llcrypt unregistered [ 8113.294737] Key type ._llcrypt unregistered [ 8131.699439] Key type ._llcrypt registered [ 8131.704255] Key type .llcrypt registered [ 8131.829589] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8133.104570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8133.253815] alg: No test for adler32 (adler32-zlib) [ 8134.321254] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 8134.556862] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 8136.223309] Key type lgssc registered [ 8137.526984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8145.721821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8145.746906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8147.309671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8152.439607] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8156.643944] LustreError: 267388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8156.658193] LustreError: 267388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8161.763917] LustreError: 267389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8163.137697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8163.657729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8169.170901] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8177.407966] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8184.253231] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8191.875380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.457700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8200.353647] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8209.099694] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8217.083625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8232.431405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8232.434196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8232.446943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8232.461749] Lustre: Skipped 1 previous similar message [ 8232.478601] Lustre: Skipped 1 previous similar message [ 8232.634268] Lustre: server umount lustre-OST0000 complete [ 8237.544188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.563783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8237.571706] Lustre: Skipped 1 previous similar message [ 8242.657381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8242.666666] Lustre: Skipped 1 previous similar message [ 8243.019892] Lustre: server umount lustre-MDT0000 complete [ 8246.802536] LustreError: 267368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629534 with bad export cookie 10471214048106457961 [ 8246.805164] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8246.816366] LustreError: 267368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8247.240753] Lustre: server umount lustre-MDT0001 complete [ 8258.329991] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8261.043804] Key type lgssc unregistered [ 8261.385225] LNet: 271084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8261.392520] LNetError: 271084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8261.408227] LNet: Removed LNI 192.168.204.129@tcp [ 8262.220534] Key type .llcrypt unregistered [ 8262.223368] Key type ._llcrypt unregistered [ 8286.597960] Key type ._llcrypt registered [ 8286.600179] Key type .llcrypt registered [ 8286.699355] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8287.836735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8287.962164] alg: No test for adler32 (adler32-zlib) [ 8289.018883] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 8289.258257] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 8290.967181] Key type lgssc registered [ 8292.067988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8300.047332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8300.097904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8301.610321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8306.003331] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8311.788703] LustreError: 272514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8311.811097] LustreError: 272514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8314.896541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8315.293848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8320.144540] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8330.141791] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8337.556268] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8345.574392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8346.008200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8347.069668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8353.071161] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8361.502933] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8390.116674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8390.117995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8390.137574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8390.154768] Lustre: Skipped 1 previous similar message [ 8390.170426] Lustre: Skipped 1 previous similar message [ 8395.237653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8395.250342] Lustre: Skipped 1 previous similar message [ 8396.531148] Lustre: server umount lustre-OST0000 complete [ 8402.401194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8402.411780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.430184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8405.472443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8405.478346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8405.497553] Lustre: Skipped 1 previous similar message [ 8407.649716] Lustre: server umount lustre-MDT0000 complete [ 8410.599155] LustreError: 272510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8410.622909] LustreError: 272510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8411.658459] LustreError: 272496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629699 with bad export cookie 630900993765651688 [ 8411.667352] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8411.672755] LustreError: 272496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8412.196491] Lustre: server umount lustre-MDT0001 complete [ 8425.657496] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8428.538674] Key type lgssc unregistered [ 8428.856489] LNet: 276821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8428.864804] LNetError: 276821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8428.891389] LNet: Removed LNI 192.168.204.129@tcp [ 8429.894273] Key type .llcrypt unregistered [ 8429.900448] Key type ._llcrypt unregistered [ 8450.000203] Key type ._llcrypt registered [ 8450.003743] Key type .llcrypt registered [ 8450.201677] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8451.702699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8451.895546] alg: No test for adler32 (adler32-zlib) [ 8453.076477] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 8453.344662] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 8455.009299] Key type lgssc registered [ 8456.703022] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8466.904687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8466.934794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.628207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8473.386754] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8479.208929] LustreError: 278057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8479.241405] LustreError: 278057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8483.788595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.493907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.740811] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8502.975960] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8511.972304] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8520.396298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8520.773016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8528.389863] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8539.270417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8539.461106] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8554.977222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8554.983976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8554.996177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8555.009983] Lustre: Skipped 1 previous similar message [ 8555.036420] Lustre: Skipped 1 previous similar message [ 8560.105736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8560.116577] Lustre: Skipped 1 previous similar message [ 8560.465264] Lustre: server umount lustre-OST0000 complete [ 8565.218319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8565.242194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8570.337324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8570.349317] Lustre: Skipped 2 previous similar messages [ 8575.456859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8575.469059] Lustre: Skipped 1 previous similar message [ 8579.039486] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8579.415706] Lustre: server umount lustre-MDT0000 complete [ 8580.578900] LustreError: 278037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8580.615059] LustreError: 278037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8583.602338] LustreError: 280805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629871 with bad export cookie 670944803593786935 [ 8583.615607] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8583.621856] LustreError: 280805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8584.018144] Lustre: server umount lustre-MDT0001 complete [ 8597.675926] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8600.973566] Key type lgssc unregistered [ 8601.355729] LNet: 281761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8601.361838] LNetError: 281761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8601.391491] LNet: Removed LNI 192.168.204.129@tcp [ 8602.278384] Key type .llcrypt unregistered [ 8602.280360] Key type ._llcrypt unregistered [ 8617.385349] Key type ._llcrypt registered [ 8617.391562] Key type .llcrypt registered [ 8617.522503] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8638.237919] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 8655.949736] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8657.354686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8657.541327] alg: No test for adler32 (adler32-zlib) [ 8658.704951] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 8659.002735] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 8660.759194] Key type lgssc registered [ 8662.366726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8705.770454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8705.810950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8707.119769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8707.146100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8707.231957] Lustre: lustre-MDT0000: new disk, initializing [ 8707.297040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.314349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8712.907987] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8725.182346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8725.372676] Lustre: 285311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8725.491706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8725.529461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8725.539316] Lustre: Skipped 1 previous similar message [ 8725.673370] Lustre: lustre-MDT0001: new disk, initializing [ 8725.859417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8725.925676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8725.937363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8731.353319] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8743.393310] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8750.073773] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8757.805273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8758.200795] Lustre: lustre-OST0000: new disk, initializing [ 8758.205771] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8758.213281] Lustre: 287285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8758.300816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.479286] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8766.057612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8766.084381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8766.249213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8777.429855] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8783.132757] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8783.342208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8789.285465] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8789.524496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8792.033133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8792.047458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.057375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8792.098893] Lustre: Skipped 1 previous similar message [ 8797.154386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8797.169589] Lustre: Skipped 2 previous similar messages [ 8802.285813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8802.303299] Lustre: Skipped 1 previous similar message [ 8805.855221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8806.119632] Lustre: server umount lustre-OST0000 complete [ 8812.519851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8812.528475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8812.543431] Lustre: Skipped 1 previous similar message [ 8822.752482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8822.763245] Lustre: Skipped 4 previous similar messages [ 8824.288566] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8824.598594] Lustre: server umount lustre-MDT0000 complete [ 8828.612240] LustreError: 285303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630116 with bad export cookie 12970861868171090257 [ 8828.613201] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8828.625343] LustreError: 285303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8829.076962] Lustre: server umount lustre-MDT0001 complete [ 8838.857838] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:42:03 (1788630123) [ 8845.866229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8846.251702] LustreError: 289699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.277485] LustreError: 289699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8846.333625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.432346] LustreError: 289700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8852.249920] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8855.585563] LustreError: 289699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8860.652559] LustreError: 289700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8863.042128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8863.529957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8869.316249] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8878.570844] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8884.850794] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8893.680115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8894.010224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8902.896360] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8914.470054] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8928.738969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8928.744935] LustreError: Skipped 1 previous similar message [ 8928.747501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.757099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8930.791469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8930.808051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.968579] Lustre: server umount lustre-OST0000 complete [ 8940.514542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8940.524034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.841897] Lustre: server umount lustre-MDT0000 complete [ 8946.145766] LustreError: 289700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.182546] LustreError: 289700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8949.111047] LustreError: 291543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630236 with bad export cookie 12970861868171091139 [ 8949.114627] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8949.122360] LustreError: 291543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8949.619385] Lustre: server umount lustre-MDT0001 complete [ 8964.892704] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8967.678808] Key type lgssc unregistered [ 8967.919447] LNet: 293239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8967.939572] LNetError: 293239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8969.000577] LNet: Removed LNI 192.168.204.129@tcp [ 8969.837442] Key type .llcrypt unregistered [ 8969.839656] Key type ._llcrypt unregistered [ 8999.764951] Key type ._llcrypt registered [ 8999.769860] Key type .llcrypt registered [ 8999.916961] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 9017.275513] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9019.149280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9019.411236] alg: No test for adler32 (adler32-zlib) [ 9021.023220] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 9021.392936] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9023.207199] Key type lgssc registered [ 9024.688511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9079.832329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9079.871451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.300630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9081.375441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9081.488518] Lustre: lustre-MDT0000: new disk, initializing [ 9081.590951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.629274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9086.877735] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9098.061914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.196247] Lustre: 297064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9098.250676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9098.258865] Lustre: Skipped 1 previous similar message [ 9098.349363] Lustre: lustre-MDT0001: new disk, initializing [ 9098.457047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9098.498961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9098.511162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9103.551875] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9114.810979] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9120.927441] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9129.594602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9129.944443] Lustre: lustre-OST2710: new disk, initializing [ 9129.954442] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9129.980984] Lustre: 299033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9130.098887] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9137.546793] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9139.272039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9139.288174] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9139.567595] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9149.403171] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9158.632560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9158.947774] Lustre: lustre-OST03e8: new disk, initializing [ 9158.957538] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9158.972446] Lustre: 300098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9159.098051] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9167.007257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9167.020118] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9167.150803] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9167.276246] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9177.847095] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9186.868489] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9187.094708] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9192.495619] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9192.811830] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9197.342345] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9197.544626] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9202.413635] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9202.682175] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9208.290330] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9208.306019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9208.339889] Lustre: Skipped 3 previous similar messages [ 9208.356170] Lustre: Skipped 3 previous similar messages [ 9213.419713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9213.431379] Lustre: Skipped 3 previous similar messages [ 9214.401941] Lustre: server umount lustre-MDT0000 complete [ 9218.437447] LustreError: 297057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630506 with bad export cookie 15572085681975084506 [ 9218.441420] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9218.450517] LustreError: 297057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9218.530196] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9218.551887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9218.559715] Lustre: Skipped 1 previous similar message [ 9218.947205] Lustre: server umount lustre-MDT0001 complete [ 9225.431350] Lustre: server umount lustre-OST2710 complete [ 9229.540459] Lustre: server umount lustre-OST03e8 complete [ 9248.847344] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 9257.920870] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9300.047753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9300.566655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9300.602395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9300.709227] Lustre: lustre-MDT0000: new disk, initializing [ 9300.894302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9300.962583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9306.598642] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9318.479845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.585785] Lustre: 305345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9318.633656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9318.641845] Lustre: Skipped 1 previous similar message [ 9318.703861] Lustre: lustre-MDT0001: new disk, initializing [ 9318.794645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9318.828276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9318.842554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9324.171395] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9335.121155] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9340.885160] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9347.724687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9347.917241] Lustre: lustre-OST0000: new disk, initializing [ 9347.921556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9347.926681] Lustre: 307318:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9348.016832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9353.282471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9353.299778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9353.408332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9354.533989] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9365.104632] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9370.017787] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9370.295391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9375.337802] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9375.549531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9379.299888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9379.309403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.313484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9379.332115] Lustre: Skipped 2 previous similar messages [ 9383.756785] Lustre: server umount lustre-OST0000 complete [ 9389.536758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9389.549241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9389.556779] Lustre: Skipped 1 previous similar message [ 9389.565432] Lustre: Skipped 1 previous similar message [ 9394.436255] Lustre: server umount lustre-MDT0000 complete [ 9394.657356] LustreError: 305355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9394.669750] LustreError: 305355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9398.302357] LustreError: 305338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630686 with bad export cookie 15572085681975086200 [ 9398.319266] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9398.333291] LustreError: 305338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9398.719805] Lustre: server umount lustre-MDT0001 complete [ 9408.886876] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:51:34 (1788630694) [ 9410.589317] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9412.603266] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:51:37 (1788630697) [ 9416.038606] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9444.018485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9444.292815] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9444.330542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9444.588589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9449.365448] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9458.864558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.005889] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9459.026504] Lustre: 311177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9465.164956] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9486.576177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9486.758543] Lustre: Denying initial registration attempt from nid 192.168.204.129@tcp, specified as failover [ 9486.764431] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9486.769685] LustreError: 312634:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9486.773443] LustreError: 312634:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9486.781356] LustreError: 312634:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9486.844399] Lustre: server umount lustre-OST0000 complete [ 9486.846876] LustreError: 312634:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9495.015122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9495.029351] LustreError: Skipped 1 previous similar message [ 9495.034365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9495.046955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9495.060344] Lustre: Skipped 1 previous similar message [ 9499.737428] Lustre: server umount lustre-MDT0000 complete [ 9502.689471] LustreError: 311190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9502.700207] LustreError: 311190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9503.732424] LustreError: 311170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630791 with bad export cookie 15572085681975087082 [ 9503.737159] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9503.738158] LustreError: 311170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9504.112651] Lustre: server umount lustre-MDT0001 complete [ 9514.453489] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 9516.962892] Key type lgssc unregistered [ 9517.239949] LNet: 313754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9517.249868] LNetError: 313754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9517.272447] LNet: Removed LNI 192.168.204.129@tcp [ 9518.023544] Key type .llcrypt unregistered [ 9518.026082] Key type ._llcrypt unregistered [ 9530.235596] Key type ._llcrypt registered [ 9530.242396] Key type .llcrypt registered [ 9530.376582] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9531.675825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9531.741387] alg: No test for adler32 (adler32-zlib) [ 9532.780325] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 9533.068545] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9534.719377] Key type lgssc registered [ 9535.781903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9584.716312] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9595.058741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9595.078770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9596.366605] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9596.392260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9596.584183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9601.909871] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9612.879447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.030676] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9613.083580] Lustre: 316518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9613.390285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9618.294415] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9627.188896] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9634.839714] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9656.774391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9657.266864] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9657.494428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.435701] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9676.275992] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9681.376475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9681.382738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9681.383028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9681.409342] Lustre: Skipped 1 previous similar message [ 9681.434524] Lustre: Skipped 1 previous similar message [ 9686.507342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9686.514340] Lustre: Skipped 1 previous similar message [ 9691.631525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9691.647289] Lustre: Skipped 1 previous similar message [ 9693.151135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9693.377418] Lustre: server umount lustre-OST0000 complete [ 9700.321529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9700.339578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9700.355166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9701.857907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.023915] Lustre: server umount lustre-MDT0000 complete [ 9708.643880] LustreError: 316509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630996 with bad export cookie 15681198018758377427 [ 9708.647066] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9708.661783] LustreError: 316509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9709.227737] Lustre: server umount lustre-MDT0001 complete [ 9721.813945] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 9724.911247] Key type lgssc unregistered [ 9725.273969] LNet: 320096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9725.286878] LNetError: 320096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9725.317842] LNet: Removed LNI 192.168.204.129@tcp [ 9726.628591] Key type .llcrypt unregistered [ 9726.630908] Key type ._llcrypt unregistered [ 9755.797690] Key type ._llcrypt registered [ 9755.799198] Key type .llcrypt registered [ 9755.998336] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9757.332562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9757.470335] alg: No test for adler32 (adler32-zlib) [ 9758.581832] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [ 9758.962773] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9760.728915] Key type lgssc registered [ 9762.218559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9771.600753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9771.632902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.145707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9778.283864] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9783.266933] LustreError: 321509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9783.285058] LustreError: 321509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9788.246846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.395566] LustreError: 321510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9788.717124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9793.678654] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9801.683254] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9808.053590] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9817.133433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9817.458949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9824.558353] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9832.758300] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9844.216787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9844.226051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9844.234582] Lustre: Skipped 1 previous similar message [ 9844.782472] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 9844.787185] Lustre: Skipped 2 previous similar messages [ 9847.743498] Lustre: server umount lustre-MDT0000 complete [ 9849.332234] LustreError: 321510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9849.359134] LustreError: 321510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9852.000069] LustreError: 321490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631139 with bad export cookie 9773459877096463205 [ 9852.020113] LustreError: 321490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9852.023806] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9854.433230] LustreError: 321505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9854.436100] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9854.473769] LustreError: 321505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9854.510386] Lustre: Skipped 1 previous similar message [ 9854.522207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9857.441355] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9857.450901] Lustre: Skipped 1 previous similar message [ 9858.567696] Lustre: server umount lustre-MDT0001 complete [ 9866.231652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9878.600741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9896.431677] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87a24df9e8de38b5 [ 9896.444745] Lustre: MGC192.168.204.129@tcp: Connection restored to 0@lo (at 0@lo) [ 9896.844849] LustreError: 323343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.871737] LustreError: 323343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9896.958587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9897.059995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9901.622236] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9907.170464] LustreError: 325108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9907.189908] LustreError: 325108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9910.989414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.400191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9912.430876] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9915.961796] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9924.527076] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9926.753332] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9934.606487] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9936.659899] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9943.525712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.545556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9948.135856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9948.148987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9949.911187] Lustre: server umount lustre-OST0000 complete [ 9957.345188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9957.357252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9957.376945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9957.379263] Lustre: Skipped 2 previous similar messages [ 9961.143806] Lustre: server umount lustre-MDT0000 complete [ 9964.000781] LustreError: 325112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9964.022177] LustreError: 325112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9965.627424] LustreError: 321491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631253 with bad export cookie 9773459877096470709 [ 9965.630282] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9965.661754] LustreError: 321491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9966.207747] Lustre: server umount lustre-MDT0001 complete [ 9979.785230] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 9982.920431] Key type lgssc unregistered [ 9983.309857] LNet: 327791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9983.319042] LNetError: 327791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9983.337474] LNet: Removed LNI 192.168.204.129@tcp [ 9984.437489] Key type .llcrypt unregistered [ 9984.439214] Key type ._llcrypt unregistered [10012.239336] Key type ._llcrypt registered [10012.241463] Key type .llcrypt registered [10012.364339] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10013.427950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10013.441525] alg: No test for adler32 (adler32-zlib) [10014.614818] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [10014.930471] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [10016.631171] Key type lgssc registered [10017.759977] Lustre: Echo OBD driver; http://www.lustre.org/ [10025.774300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10025.813361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.303746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10031.542322] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10036.705048] LustreError: 329206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.724838] LustreError: 329206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10041.177804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10041.558335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10046.290539] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10054.290866] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10061.295580] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10066.860228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10066.873953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10066.887925] Lustre: Skipped 1 previous similar message [10072.037907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10072.043696] Lustre: Skipped 1 previous similar message [10077.161736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10077.173632] Lustre: Skipped 1 previous similar message [10079.199175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10079.515587] Lustre: server umount lustre-MDT0000 complete [10082.276197] LustreError: 329201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.302884] LustreError: 329201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10083.434517] LustreError: 329186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631371 with bad export cookie 16746547603069707704 [10083.442109] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10083.452034] LustreError: 329186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10083.880683] Lustre: server umount lustre-MDT0001 complete [10100.429566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10100.903339] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10100.948850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10101.238633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10106.488501] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10116.254372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.377512] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10116.418984] Lustre: 331795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10116.613311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10120.893728] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10129.307311] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10135.980846] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10145.903142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10146.099484] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10146.116369] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10146.129677] LustreError: 333679:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10146.136404] LustreError: 333679:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10146.140365] LustreError: 333679:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10146.224392] Lustre: server umount lustre-OST0000 complete [10146.234532] LustreError: 333679:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10158.645727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10158.932102] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10159.205348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10166.017528] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10166.422821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10173.855477] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10182.622138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10182.718292] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10182.815135] Lustre: lustre-OST0001: new disk, initializing [10182.821575] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10182.832968] Lustre: 335222:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10182.934135] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10188.431785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10188.451118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10188.544110] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10189.405411] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10197.574263] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10204.136719] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10204.137355] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10204.143377] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10204.168027] Lustre: Skipped 2 previous similar messages [10209.268315] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10209.275369] Lustre: Skipped 2 previous similar messages [10215.392276] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10215.494506] Lustre: server umount lustre-OST0001 complete [10219.489274] LustreError: 334145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10219.499197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10219.510911] LustreError: 334145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10219.529332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10219.544137] Lustre: Skipped 3 previous similar messages [10224.610595] LustreError: 334147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10224.635552] LustreError: 334147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10229.752364] LustreError: 334146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10229.779915] LustreError: 334146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10233.311157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10233.474451] Lustre: server umount lustre-OST0000 complete [10239.457482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10239.470035] LustreError: Skipped 1 previous similar message [10239.478301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10239.500610] Lustre: Skipped 1 previous similar message [10239.508070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10239.520419] Lustre: Skipped 4 previous similar messages [10244.428757] Lustre: server umount lustre-MDT0000 complete [10245.094234] LustreError: 331806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10245.122409] LustreError: 331806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10247.846303] LustreError: 331788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631535 with bad export cookie 16746547603069708257 [10247.850589] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10247.858711] LustreError: 331788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10248.176914] Lustre: server umount lustre-MDT0001 complete [10259.148489] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [10261.818764] Key type lgssc unregistered [10262.078387] LNet: 337179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10262.084148] LNetError: 337179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10262.095215] LNet: Removed LNI 192.168.204.129@tcp [10263.028669] Key type .llcrypt unregistered [10263.030942] Key type ._llcrypt unregistered [10333.242425] Key type ._llcrypt registered [10333.244148] Key type .llcrypt registered [10333.371295] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [10350.226211] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10351.808297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10351.849015] alg: No test for adler32 (adler32-zlib) [10352.895644] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [10353.193329] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [10354.879257] Key type lgssc registered [10356.148860] Lustre: Echo OBD driver; http://www.lustre.org/ [10392.832164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10392.865284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.140401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10394.159410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10394.214994] Lustre: lustre-MDT0000: new disk, initializing [10394.307853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10394.322299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10398.420786] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10408.445153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10408.567945] Lustre: 342116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10408.639161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10408.643206] Lustre: Skipped 1 previous similar message [10408.702428] Lustre: lustre-MDT0001: new disk, initializing [10408.828108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10408.880401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10408.892744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10413.623195] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10423.095379] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10428.871093] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10435.437736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.682910] Lustre: lustre-OST0000: new disk, initializing [10435.686363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10435.690595] Lustre: 344084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10435.771425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10442.841469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10442.848609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10443.026376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10443.378714] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10455.737989] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10460.667548] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10460.908837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10466.057871] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10466.355682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10468.837312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10468.847125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10468.855634] Lustre: Skipped 1 previous similar message [10468.858171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10473.966716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10473.973989] Lustre: Skipped 2 previous similar messages [10474.751566] Lustre: server umount lustre-OST0000 complete [10480.614760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10480.622570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10480.635491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10484.195270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10484.199025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10486.145155] Lustre: server umount lustre-MDT0000 complete [10490.924030] LustreError: 342106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631778 with bad export cookie 5589135696154033639 [10490.929786] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10490.939729] LustreError: 342106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10491.731139] Lustre: server umount lustre-MDT0001 complete [10502.334675] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:09:47 (1788631787) [10510.733726] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:09:56 (1788631796) [10519.375385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10519.734903] LustreError: 346841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10519.753102] LustreError: 346841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10519.807509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10524.340491] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10525.154349] LustreError: 346842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.284267] LustreError: 346841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10532.931300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10533.380889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10538.951934] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10547.289581] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10553.355558] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10560.395166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10560.756756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10566.759274] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10574.287336] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10587.198681] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10589.614661] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10591.571170] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10593.804076] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10596.732062] Lustre: Failing over lustre-MDT0000 [10597.438666] LustreError: 346836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10597.489619] Lustre: server umount lustre-MDT0000 complete [10599.914623] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10599.916096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10599.941031] Lustre: Skipped 2 previous similar messages [10602.508617] LustreError: 348682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.532863] LustreError: 348682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10612.760854] LustreError: 346838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10612.780787] LustreError: 346838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10616.287108] Lustre: 340336:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788631887/real 1788631887] req@00000000171b4749 x1875516210617984/t0(0) o400->MGC192.168.204.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788631903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10616.299855] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10617.603758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10626.529904] LustreError: 340335:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bc405d8a x1875516210623744/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10626.927937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10627.011179] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10629.141066] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10631.763443] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10632.173537] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10632.208326] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10632.257364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10642.373691] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10644.061161] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10648.290423] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10650.358769] Lustre: Failing over lustre-MDT0000 [10650.670995] Lustre: server umount lustre-MDT0000 complete [10652.647320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10652.674894] Lustre: Skipped 1 previous similar message [10652.675688] LustreError: 350695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10652.706639] LustreError: 350695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [10668.903126] Lustre: 340336:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788631940/real 1788631940] req@000000006024c1c1 x1875516210639744/t0(0) o400->MGC192.168.204.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788631956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10668.948276] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10669.772136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.271633] LustreError: 340335:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000034ce5137 x1875516210646016/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10679.632709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10679.706331] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10680.337359] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10684.267274] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10684.917479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10684.924130] Lustre: Skipped 2 previous similar messages [10684.971934] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10685.023321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10694.801489] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10696.635301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10705.385967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10705.389535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10705.392470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10705.392482] Lustre: Skipped 1 previous similar message [10705.400768] Lustre: Skipped 3 previous similar messages [10710.404813] Lustre: server umount lustre-MDT0000 complete [10710.513873] LustreError: 346841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10710.532045] LustreError: 346841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [10714.268622] LustreError: 346820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632001 with bad export cookie 5589135696154036383 [10714.279901] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10714.665688] Lustre: server umount lustre-MDT0001 complete [10728.990219] Lustre: server umount lustre-OST0000 complete [10746.223949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10746.901121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10751.476816] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10760.649760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10765.718871] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10774.388525] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10781.436481] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10789.316148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10791.744621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10796.109427] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10804.688875] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10816.481871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10816.482790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10816.494780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10816.510277] Lustre: Skipped 1 previous similar message [10816.514817] Lustre: Skipped 2 previous similar messages [10822.102224] Lustre: server umount lustre-OST0000 complete [10836.965153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10836.979019] Lustre: Skipped 8 previous similar messages [10840.031516] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10840.323502] Lustre: server umount lustre-MDT0000 complete [10842.080484] LustreError: 353200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.109742] LustreError: 353200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10844.540739] LustreError: 353184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632132 with bad export cookie 5589135696154037104 [10844.548894] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10844.557313] LustreError: 353184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10844.934630] Lustre: server umount lustre-MDT0001 complete [10858.831838] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [10862.068559] Key type lgssc unregistered [10862.412674] LNet: 356738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10862.424730] LNetError: 356738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10862.440604] LNet: Removed LNI 192.168.204.129@tcp [10863.219182] Key type .llcrypt unregistered [10863.221477] Key type ._llcrypt unregistered [10892.686603] Key type ._llcrypt registered [10892.696317] Key type .llcrypt registered [10892.866714] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10894.158673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10894.174106] alg: No test for adler32 (adler32-zlib) [10895.251694] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [10895.528705] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [10897.234379] Key type lgssc registered [10898.379638] Lustre: Echo OBD driver; http://www.lustre.org/ [10906.013798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10906.035419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10907.571450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10913.126386] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10917.857806] LustreError: 358250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10917.873877] LustreError: 358250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10922.695242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10922.978597] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10923.161437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10927.926900] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10936.897686] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10943.717729] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10952.123921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10952.517134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10953.584935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10959.335771] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10968.576461] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10983.394873] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10983.397018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.419377] Lustre: Skipped 2 previous similar messages [10983.451261] Lustre: Skipped 2 previous similar messages [10988.477726] Lustre: server umount lustre-MDT0000 complete [10988.530020] LustreError: 360702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10988.546881] LustreError: 360702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10992.381109] LustreError: 358231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632280 with bad export cookie 16160535147346600069 [10992.393421] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10992.403898] LustreError: 358231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10992.772240] Lustre: server umount lustre-MDT0001 complete [11007.655148] Lustre: server umount lustre-OST0000 complete [11023.302792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11023.853358] LustreError: 361988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11023.951780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11028.964292] LustreError: 361989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11028.980150] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11034.088800] LustreError: 361988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11038.867857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11039.206169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11039.427117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11044.560253] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11054.024988] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11061.246579] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11068.950854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.268739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11076.064564] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11084.418788] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11093.886629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11109.344581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.349331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11109.378585] Lustre: Skipped 2 previous similar messages [11109.395161] Lustre: Skipped 2 previous similar messages [11114.467495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11114.480560] Lustre: Skipped 1 previous similar message [11115.601458] Lustre: server umount lustre-MDT0000 complete [11119.420432] LustreError: 361968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632407 with bad export cookie 16160535147346601189 [11119.432650] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11119.439904] LustreError: 361968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11119.600713] LustreError: 361985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11119.601970] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11120.076576] Lustre: server umount lustre-MDT0001 complete [11126.202655] Lustre: server umount lustre-OST0000 complete [11143.182519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11143.687329] LustreError: 365923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11143.755337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11148.294175] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11158.186094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11163.382947] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11172.852734] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11179.694238] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11187.697940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11188.064643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11188.070918] Lustre: Skipped 1 previous similar message [11189.107524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11195.047859] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11203.623921] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11208.672399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11208.679255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11208.694408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11208.698761] Lustre: Skipped 1 previous similar message [11208.715508] Lustre: Skipped 3 previous similar messages [11214.873445] Lustre: server umount lustre-OST0000 complete [11219.937257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11219.958064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11225.819833] Lustre: server umount lustre-MDT0000 complete [11229.153520] LustreError: 365923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11229.175156] LustreError: 365923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11229.861871] LustreError: 365905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632517 with bad export cookie 16160535147346602211 [11229.870537] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11229.883966] LustreError: 365905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11230.445815] Lustre: server umount lustre-MDT0001 complete [11246.778923] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [11250.614478] Key type lgssc unregistered [11251.139181] LNet: 369455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11251.157691] LNetError: 369455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11251.187883] LNet: Removed LNI 192.168.204.129@tcp [11252.488524] Key type .llcrypt unregistered [11252.490300] Key type ._llcrypt unregistered [11284.118983] Key type ._llcrypt registered [11284.120661] Key type .llcrypt registered [11284.269945] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [11286.315364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11286.402249] alg: No test for adler32 (adler32-zlib) [11287.637832] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [11288.069078] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [11289.855193] Key type lgssc registered [11290.906561] Lustre: Echo OBD driver; http://www.lustre.org/ [11299.597887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11299.645570] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11299.660471] LustreError: 370893:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11299.686516] LustreError: 370893:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11299.695940] LustreError: 370893:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11299.705462] LustreError: 370893:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11299.713845] LustreError: 370893:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11307.770486] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11307.794882] LustreError: 371291:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11307.824170] LustreError: 371291:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11307.845070] LustreError: 371291:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11307.861140] LustreError: 371291:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11307.882957] LustreError: 371291:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11330.843694] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [11334.126215] Key type lgssc unregistered [11334.519301] LNet: 372294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11334.526634] LNetError: 372294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11334.557759] LNet: Removed LNI 192.168.204.129@tcp [11335.509361] Key type .llcrypt unregistered [11335.548076] Key type ._llcrypt unregistered [11357.763828] Key type ._llcrypt registered [11357.773094] Key type .llcrypt registered [11357.959807] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [11374.149359] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [11375.303488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11375.315990] alg: No test for adler32 (adler32-zlib) [11376.348773] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [11376.667490] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [11378.367819] Key type lgssc registered [11379.503320] Lustre: Echo OBD driver; http://www.lustre.org/ [11417.322526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11417.362558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11418.576156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11418.598272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11418.662205] Lustre: lustre-MDT0000: new disk, initializing [11418.722790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11418.740233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11423.593570] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11434.679818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11434.903151] Lustre: 375451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11435.020756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11435.031032] Lustre: Skipped 1 previous similar message [11435.181593] Lustre: lustre-MDT0001: new disk, initializing [11435.319431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11435.357378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11435.372739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11440.015603] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11450.667104] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11456.470975] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11465.180193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11465.521179] Lustre: lustre-OST0000: new disk, initializing [11465.534772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11465.552590] Lustre: 377415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11465.681938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11472.510062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11472.528537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11472.652055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11473.868576] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11484.494966] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11490.098423] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11490.318530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11494.977103] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11495.170592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11498.469623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11498.470308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11498.487296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11503.579422] Lustre: server umount lustre-OST0000 complete [11508.543279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11508.564469] Lustre: Skipped 2 previous similar messages [11508.576375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11508.589032] Lustre: Skipped 2 previous similar messages [11513.837147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11513.854978] Lustre: Skipped 1 previous similar message [11515.050033] Lustre: server umount lustre-MDT0000 complete [11519.804181] LustreError: 375443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632807 with bad export cookie 3600572225048517359 [11519.833209] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11520.191776] Lustre: server umount lustre-MDT0001 complete [11532.427581] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11534.942914] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:26:59 (1788632819) [11542.397220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11542.885426] LustreError: 379876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11542.915027] LustreError: 379876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11542.992850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11548.083835] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11548.129403] LustreError: 379877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11553.251426] LustreError: 379876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.124992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11557.478659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11562.256212] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11571.313705] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11576.936612] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11584.269804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11584.717976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11592.849393] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11605.644879] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11614.373106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11614.608502] Lustre: lustre-OST0001: new disk, initializing [11614.617685] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11614.636718] Lustre: 382713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11614.757923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11622.993559] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11624.651248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11624.673457] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11624.875626] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11634.767774] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11640.288974] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11640.322836] LustreError: Skipped 1 previous similar message [11640.324161] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11640.328697] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11640.360560] Lustre: Skipped 1 previous similar message [11645.438762] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11645.479594] Lustre: Skipped 3 previous similar messages [11646.186075] Lustre: server umount lustre-OST0001 complete [11646.992781] LustreError: 381699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11652.109586] LustreError: 381697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11652.171342] LustreError: 381697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11660.785390] LustreError: 383148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11660.821827] LustreError: 383148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11669.984886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11669.986323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11669.999264] LustreError: Skipped 1 previous similar message [11670.022770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11670.033333] Lustre: Skipped 1 previous similar message [11673.390097] Lustre: server umount lustre-OST0000 complete [11680.225234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11680.240075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11680.257790] Lustre: Skipped 1 previous similar message [11684.089191] Lustre: server umount lustre-MDT0000 complete [11686.372503] LustreError: 379871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.411693] LustreError: 379871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11687.997432] LustreError: 379857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632975 with bad export cookie 3600572225048518241 [11688.003688] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11688.014694] LustreError: 379857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11694.773538] Lustre: server umount lustre-MDT0001 complete [11707.397067] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [11710.437551] Key type lgssc unregistered [11710.810624] LNet: 384704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11710.824826] LNetError: 384704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11710.850648] LNet: Removed LNI 192.168.204.129@tcp [11711.703752] Key type .llcrypt unregistered [11711.705887] Key type ._llcrypt unregistered [11771.069818] Key type ._llcrypt registered [11771.074065] Key type .llcrypt registered [11771.289311] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [11772.299960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11772.576479] alg: No test for adler32 (adler32-zlib) [11773.684163] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [11773.996604] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [11775.727670] Key type lgssc registered [11777.118253] Lustre: Echo OBD driver; http://www.lustre.org/ [11785.326980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11785.360861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11786.716913] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11786.742596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11787.087229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11791.835336] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11800.897504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.074224] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11801.113299] Lustre: 387502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11801.419850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11806.285699] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11814.287460] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11819.651204] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11822.050876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11822.053521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.068233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11822.078867] Lustre: Skipped 1 previous similar message [11822.096107] Lustre: Skipped 1 previous similar message [11827.168530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11827.174892] Lustre: Skipped 1 previous similar message [11827.604390] Lustre: server umount lustre-MDT0000 complete [11831.123114] LustreError: 387494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633118 with bad export cookie 13772766835669890867 [11831.129224] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11831.135303] LustreError: 387494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11831.448343] Lustre: server umount lustre-MDT0001 complete [11842.182736] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:32:07 (1788633127) [11849.267936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11849.800249] LustreError: 390015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11849.822423] LustreError: 390015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11849.869750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11854.960446] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11855.339118] LustreError: 390016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11860.451138] LustreError: 390015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.965262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11864.342908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11869.209339] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11877.340671] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11883.102464] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11890.555079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11890.941378] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11891.281952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11898.502752] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11907.622814] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11915.357268] Lustre: Permanently deactivating lustre-OST0000 [11915.379406] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11920.923550] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11923.724614] LustreError: 392955:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11923.732964] LustreError: 392955:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11923.738221] LustreError: 392955:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11927.031773] LustreError: 393004:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11927.040056] LustreError: 393004:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11935.443382] Lustre: server umount lustre-OST0000 complete [11938.584675] LustreError: 393253:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11938.590212] LustreError: 393253:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11941.344723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11941.353450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11941.373905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11942.887652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11948.003263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11948.011703] Lustre: Skipped 3 previous similar messages [11953.120674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11953.131830] Lustre: Skipped 1 previous similar message [11955.167789] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11955.362285] Lustre: server umount lustre-MDT0000 complete [11958.245359] LustreError: 390753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11958.264159] LustreError: 390753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11958.584177] LustreError: 389996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633246 with bad export cookie 13772766835669891448 [11958.592823] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11958.599341] LustreError: 389996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11969.171447] Lustre: server umount lustre-MDT0001 complete [11980.540906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11987.077675] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11990.899886] LustreError: 394417:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11999.751659] Lustre: 394756:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.129@tcp [12003.178793] Lustre: server umount MGS complete [12017.786413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12018.410021] LustreError: 395481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.441724] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12018.446957] Lustre: Skipped 1 previous similar message [12018.499785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12023.995798] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12028.904616] LustreError: 395482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.924465] LustreError: 395482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12033.978780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12034.188415] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12039.237206] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12047.342732] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12054.229926] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12056.322257] Lustre: Permanently reactivating lustre-OST0000 [12056.363704] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12056.374832] Lustre: Skipped 2 previous similar messages [12084.712157] LustreError: 397263:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12094.400901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.789879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12094.798800] Lustre: Skipped 1 previous similar message [12102.108469] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12110.762881] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12120.547637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12120.553047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12120.561609] LustreError: Skipped 1 previous similar message [12120.588568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12123.014257] Lustre: server umount lustre-OST0000 complete [12130.795187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12130.828757] Lustre: Skipped 2 previous similar messages [12133.584819] Lustre: server umount lustre-MDT0000 complete [12135.908383] LustreError: 395481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.934249] LustreError: 395481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12138.004865] LustreError: 395462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633425 with bad export cookie 13772766835669892498 [12138.014138] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12138.029395] LustreError: 395462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12138.465069] Lustre: server umount lustre-MDT0001 complete [12151.383640] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [12154.363608] Key type lgssc unregistered [12154.641130] LNet: 399848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12154.657396] LNetError: 399848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12154.677091] LNet: Removed LNI 192.168.204.129@tcp [12155.712144] Key type .llcrypt unregistered [12155.715564] Key type ._llcrypt unregistered [12175.745652] Key type ._llcrypt registered [12175.752042] Key type .llcrypt registered [12175.883088] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [12194.286587] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12195.473606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12195.549580] alg: No test for adler32 (adler32-zlib) [12196.597722] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [12196.795665] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [12198.464137] Key type lgssc registered [12199.997153] Lustre: Echo OBD driver; http://www.lustre.org/ [12238.667699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12238.695976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12239.955701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12239.989825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12240.060177] Lustre: lustre-MDT0000: new disk, initializing [12240.147140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12240.163775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12243.871318] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12253.372568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.503407] Lustre: 403004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12253.582127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12253.591505] Lustre: Skipped 1 previous similar message [12253.764799] Lustre: lustre-MDT0001: new disk, initializing [12253.891547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12253.921840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12253.936970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12257.494136] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12266.579693] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12271.853394] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12278.899332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12279.192693] Lustre: lustre-OST0000: new disk, initializing [12279.197691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12279.205488] Lustre: 404969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12279.310479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12285.475774] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12289.628420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12289.643186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12289.738237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12294.687804] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12299.874179] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12300.109750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12305.002924] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12305.276675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12310.498579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12310.501482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12310.503072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12310.514740] LustreError: Skipped 1 previous similar message [12310.541892] Lustre: Skipped 1 previous similar message [12313.637636] Lustre: server umount lustre-OST0000 complete [12320.740961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12320.741458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12320.744379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12320.744387] Lustre: Skipped 1 previous similar message [12320.753876] Lustre: Skipped 1 previous similar message [12323.203392] Lustre: server umount lustre-MDT0000 complete [12326.848478] LustreError: 403797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633614 with bad export cookie 8076617319214344176 [12326.854722] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12326.859067] LustreError: 403797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12327.243379] Lustre: server umount lustre-MDT0001 complete [12336.268952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12338.108302] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:40:23 (1788633623) [12346.023250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12346.573285] LustreError: 407433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12346.602394] LustreError: 407433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12346.646256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12351.516714] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12351.969971] LustreError: 407434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12357.091724] LustreError: 407433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.737819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12361.184373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12365.681765] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12372.737741] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12379.469158] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12386.532501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12386.880628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12393.494269] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12401.257858] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12403.504303] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12406.352969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12421.611669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12421.613618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12421.615356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12421.615362] Lustre: Skipped 1 previous similar message [12421.641539] Lustre: Skipped 1 previous similar message [12424.691559] Lustre: server umount lustre-OST0000 complete [12431.840596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12431.849069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12431.876059] Lustre: Skipped 1 previous similar message [12431.902431] Lustre: Skipped 2 previous similar messages [12436.382732] Lustre: server umount lustre-MDT0000 complete [12436.960174] LustreError: 407433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12436.980323] LustreError: 407433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12440.554543] LustreError: 407415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633728 with bad export cookie 8076617319214345058 [12440.560852] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12440.573108] LustreError: 407415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12440.980555] Lustre: server umount lustre-MDT0001 complete [12452.071148] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [12455.010326] Key type lgssc unregistered [12455.266760] LNet: 411159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12455.276351] LNetError: 411159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12455.294792] LNet: Removed LNI 192.168.204.129@tcp [12456.122806] Key type .llcrypt unregistered [12456.126185] Key type ._llcrypt unregistered [12482.515627] Key type ._llcrypt registered [12482.520531] Key type .llcrypt registered [12482.727835] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [12483.164380] Key type .llcrypt unregistered [12483.167974] Key type ._llcrypt unregistered [12504.319592] Key type ._llcrypt registered [12504.321993] Key type .llcrypt registered [12504.427883] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12505.300805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12505.492680] alg: No test for adler32 (adler32-zlib) [12506.706232] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [12506.958648] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [12508.743271] Key type lgssc registered [12510.009786] Lustre: Echo OBD driver; http://www.lustre.org/ [12518.783201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12518.818632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12520.428710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12525.300042] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12530.666904] LustreError: 413365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12530.714971] LustreError: 413365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12534.577873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12535.107521] LustreError: 413364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12535.143375] LustreError: 413364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12535.170857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12540.388734] LustreError: 413359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12540.419078] LustreError: 413359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12543.407900] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12545.508289] LustreError: 413365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12545.536559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12545.543677] LustreError: 413365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12550.631768] LustreError: 413364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12550.650480] LustreError: 413364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12552.041979] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12559.656050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12560.150808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12564.787962] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12575.716891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12575.724351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12575.729765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12575.734668] Lustre: Skipped 1 previous similar message [12575.759159] Lustre: Skipped 1 previous similar message [12578.345065] Lustre: server umount lustre-OST0000 complete [12585.955064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12585.956121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12585.966586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12585.979352] Lustre: Skipped 1 previous similar message [12585.995062] Lustre: Skipped 1 previous similar message [12587.627567] Lustre: server umount lustre-MDT0000 complete [12591.050047] LustreError: 415470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633878 with bad export cookie 16202636258267125246 [12591.054983] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12591.062332] LustreError: 415470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12591.393685] Lustre: server umount lustre-MDT0001 complete [12600.398578] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [12602.989391] Key type lgssc unregistered [12603.350387] LNet: 416512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12603.368540] LNetError: 416512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12603.400293] LNet: Removed LNI 192.168.204.129@tcp [12604.200543] Key type .llcrypt unregistered [12604.204858] Key type ._llcrypt unregistered [12626.161138] Key type ._llcrypt registered [12626.162926] Key type .llcrypt registered [12626.328339] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12627.419400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12627.466045] alg: No test for adler32 (adler32-zlib) [12628.610810] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [12628.920970] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [12630.647206] Key type lgssc registered [12631.748444] Lustre: Echo OBD driver; http://www.lustre.org/ [12639.816639] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12639.851401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12655.391509] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12660.511509] LustreError: 417891:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [12687.263236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12692.832309] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12699.777950] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12706.268458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12706.805836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12706.892272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12711.624714] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12711.916672] LustreError: 417917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.954643] LustreError: 417917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12717.028451] LustreError: 417916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12720.828315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12721.200277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12725.321674] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12732.602181] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12738.714953] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12746.723257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12746.739339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12746.771325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12747.744456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12747.750721] LustreError: 417907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12747.750939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12747.793226] LustreError: 417907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12747.955477] Lustre: server umount lustre-OST0000 complete [12751.781371] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12751.796791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12751.816529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12752.867652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12752.876490] Lustre: Skipped 1 previous similar message [12757.987076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12757.996889] Lustre: Skipped 1 previous similar message [12763.104456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12763.110017] Lustre: Skipped 1 previous similar message [12766.175206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12766.462827] Lustre: server umount lustre-MDT0000 complete [12768.226281] LustreError: 418929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12768.269567] LustreError: 418929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12770.257267] LustreError: 419669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634057 with bad export cookie 7776606477064208820 [12770.263988] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12770.281155] LustreError: 419669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12770.819504] Lustre: server umount lustre-MDT0001 complete [12782.255136] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [12785.093740] Key type lgssc unregistered [12785.414524] LNet: 421460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12785.419583] LNetError: 421460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12785.433934] LNet: Removed LNI 192.168.204.129@tcp [12786.504176] Key type .llcrypt unregistered [12786.506071] Key type ._llcrypt unregistered [12813.046637] Key type ._llcrypt registered [12813.049740] Key type .llcrypt registered [12813.165862] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12814.167895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12814.243710] alg: No test for adler32 (adler32-zlib) [12815.282716] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [12815.522666] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [12817.175501] Key type lgssc registered [12818.194759] Lustre: Echo OBD driver; http://www.lustre.org/ [12826.934780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12826.977897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12828.466288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12832.349526] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12838.894362] LustreError: 422874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12838.903544] LustreError: 422874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12841.620436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.969115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12847.488887] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12855.276736] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12862.321312] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12871.513428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12872.096103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12874.196631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12879.219139] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12888.086364] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12893.158319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12893.167966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12893.188517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12895.793759] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) [12896.742295] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12899.239496] Lustre: server umount lustre-MDT0000 complete [12900.878111] LustreError: 422869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12902.312242] LustreError: 422853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.29@tcp arrived at 1788634190 with bad export cookie 672896332138174304 [12904.406878] LustreError: 422855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634192 with bad export cookie 672896332138173408 [12904.424107] LustreError: 422855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12904.432579] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12906.979201] LustreError: 422873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12906.984945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12907.017824] LustreError: 422873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12907.058450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12907.073285] Lustre: Skipped 1 previous similar message [12907.085259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12907.098789] Lustre: Skipped 3 previous similar messages [12909.987639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12909.989516] LustreError: 424088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12910.015649] LustreError: 424088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12910.662371] Lustre: server umount lustre-OST0000 complete [12915.105254] LustreError: 422873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12915.123638] LustreError: 422873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12917.592465] Lustre: server umount lustre-MDT0001 complete [12928.416750] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [12930.919843] Key type lgssc unregistered [12931.213180] LNet: 426559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12931.222184] LNetError: 426559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12931.245794] LNet: Removed LNI 192.168.204.129@tcp [12931.999165] Key type .llcrypt unregistered [12932.006084] Key type ._llcrypt unregistered [12956.751386] Key type ._llcrypt registered [12956.759197] Key type .llcrypt registered [12956.937619] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12958.735978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12958.758340] alg: No test for adler32 (adler32-zlib) [12959.926689] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [12960.262221] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [12962.007873] Key type lgssc registered [12963.192291] Lustre: Echo OBD driver; http://www.lustre.org/ [12970.513831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12970.537690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12972.208404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12976.795285] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12981.733533] LustreError: 427969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12981.756597] LustreError: 427969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12985.682872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.027671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12990.243246] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12998.002253] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13003.565665] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13010.472858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13010.843549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13016.778822] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13017.018994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13024.330599] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13031.487053] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [13032.417920] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13032.448261] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13032.451977] Lustre: Skipped 1 previous similar message [13032.479075] Lustre: Skipped 1 previous similar message [13034.215245] Lustre: server umount lustre-MDT0001 complete [13036.559773] LustreError: 427964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13042.658806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13042.661928] LustreError: 429014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13042.671117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13042.673248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13042.690251] LustreError: 429014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13047.780894] LustreError: 427970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13047.785346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13048.292873] Lustre: server umount lustre-OST0000 complete [13052.900081] LustreError: 429014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13053.789216] Lustre: server umount lustre-MDT0000 complete [13069.108880] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [13072.006840] Key type lgssc unregistered [13072.354868] LNet: 431651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13072.361863] LNetError: 431651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13073.383665] LNet: Removed LNI 192.168.204.129@tcp [13074.283246] Key type .llcrypt unregistered [13074.285450] Key type ._llcrypt unregistered [13100.240805] Key type ._llcrypt registered [13100.254691] Key type .llcrypt registered [13100.341146] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [13116.314652] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [13117.725661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13117.800571] alg: No test for adler32 (adler32-zlib) [13118.864672] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [13119.107920] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [13120.791425] Key type lgssc registered [13121.888223] Lustre: Echo OBD driver; http://www.lustre.org/ [13168.314629] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [13185.972377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13186.004765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13187.229134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13187.254433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13187.361126] Lustre: lustre-MDT0000: new disk, initializing [13187.469699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13187.494767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13191.971725] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13208.652668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13208.960479] Lustre: lustre-OST0000: new disk, initializing [13208.964514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13208.968297] Lustre: Skipped 1 previous similar message [13208.976613] Lustre: 436683:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13209.052574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13215.812609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13215.828765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13215.863824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13216.001896] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13227.230551] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13243.260987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13243.378799] Lustre: 435665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13243.474634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13243.562626] Lustre: lustre-MDT0001: new disk, initializing [13243.725131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13243.860730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13243.873224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13248.283787] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13257.951899] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13258.167663] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13263.230150] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13263.486202] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13272.038809] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13272.040146] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13272.041841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13272.089965] Lustre: Skipped 1 previous similar message [13274.703152] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [13274.716507] Lustre: Skipped 1 previous similar message [13277.154850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13277.165981] Lustre: Skipped 1 previous similar message [13279.755997] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [13284.319291] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13284.520420] Lustre: server umount lustre-MDT0001 complete [13292.536577] LustreError: 435677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13292.559320] LustreError: 435677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13297.634671] LustreError: 435676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13297.655655] LustreError: 435676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13302.776033] LustreError: 437204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13302.792910] LustreError: 437204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13307.874884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13307.876095] LustreError: 435676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13307.900087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13307.903346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13307.903354] Lustre: Skipped 2 previous similar messages [13309.236042] Lustre: server umount lustre-OST0000 complete [13313.002190] LustreError: 435677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13319.753309] Lustre: server umount lustre-MDT0000 complete [13333.899123] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [13336.844271] Key type lgssc unregistered [13337.320738] LNet: 440438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13337.329592] LNetError: 440438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13337.350535] LNet: Removed LNI 192.168.204.129@tcp [13338.474270] Key type .llcrypt unregistered [13338.477268] Key type ._llcrypt unregistered [13385.263199] Key type ._llcrypt registered [13385.267141] Key type .llcrypt registered [13385.403409] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [13385.693219] Key type .llcrypt unregistered [13385.696729] Key type ._llcrypt unregistered [13403.032969] Key type ._llcrypt registered [13403.035151] Key type .llcrypt registered [13403.140398] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [13404.303221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13404.428078] alg: No test for adler32 (adler32-zlib) [13405.518590] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [13405.777629] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [13407.487324] Key type lgssc registered [13408.707246] Lustre: Echo OBD driver; http://www.lustre.org/ [13439.139826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13439.161820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13440.448788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13440.481173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13440.550789] Lustre: lustre-MDT0000: new disk, initializing [13440.614321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13440.633760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13445.413849] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13456.369195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13456.484299] Lustre: 443830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13456.534610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13456.537793] Lustre: Skipped 1 previous similar message [13456.624451] Lustre: lustre-MDT0001: new disk, initializing [13456.737282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13456.791241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13456.816088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13460.607755] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13470.429175] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13476.444545] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13483.817255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13484.110630] Lustre: lustre-OST0000: new disk, initializing [13484.120068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13484.137551] Lustre: 445794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13484.238352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13490.640174] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13493.302956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13493.321549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13493.481338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13500.862770] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13508.066932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13508.084799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13508.114915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13509.090996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13509.096067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13509.113282] Lustre: Skipped 1 previous similar message [13509.132438] Lustre: Skipped 2 previous similar messages [13513.748936] Lustre: server umount lustre-MDT0000 complete [13517.161969] LustreError: 443820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634804 with bad export cookie 4982658081543096117 [13517.163980] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13517.171357] LustreError: 443820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13517.463908] Lustre: server umount lustre-MDT0001 complete [13531.841457] Lustre: server umount lustre-OST0000 complete [13541.722991] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:00:27 (1788634827) [13549.670163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13550.204872] LustreError: 447822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13550.227788] LustreError: 447822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13550.255230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13555.691016] LustreError: 447823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13556.380525] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13559.777235] LustreError: 447822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13564.903290] LustreError: 447823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13566.207021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13566.802582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13571.967317] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13581.923411] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13587.973219] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13607.975634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13608.241092] Lustre: Found index 0 for lustre-OST0000, updating log [13608.427492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13613.474744] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13620.714713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13621.361676] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13626.857468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13626.866780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13626.907865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13633.000380] Lustre: server umount lustre-OST0000 complete [13649.378832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13649.383832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13649.390106] Lustre: Skipped 2 previous similar messages [13649.399220] Lustre: Skipped 4 previous similar messages [13653.475082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13653.484538] Lustre: Skipped 1 previous similar message [13655.404633] Lustre: server umount lustre-MDT0000 complete [13659.200946] LustreError: 447804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634946 with bad export cookie 4982658081543097510 [13659.205790] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13659.217577] LustreError: 447804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13659.566805] Lustre: server umount lustre-MDT0001 complete [13668.969966] LNet: 1 local NIs in recovery (showing 1): 192.168.204.129@tcp [13669.124490] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:02:34 (1788634954) [14296.453361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14296.989792] LustreError: 454247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14297.075641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14302.183216] LustreError: 454248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14302.213571] LustreError: 454248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14302.932533] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [14307.300850] LustreError: 454247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14313.441090] LustreError: 454248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14313.453309] LustreError: 454248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14313.556705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14314.005675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14318.733587] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [14326.657112] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14333.330856] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14355.631650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14355.989767] Lustre: Found index 0 for lustre-OST0000, updating log [14356.436957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14360.227654] LNet: 1 local NIs in recovery (showing 1): 192.168.204.129@tcp [14363.181290] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [14371.736995] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14375.148965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14375.178783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14375.193183] Lustre: Skipped 2 previous similar messages [14380.594683] Lustre: server umount lustre-OST0000 complete [14385.314836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14385.334194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14385.347149] Lustre: Skipped 1 previous similar message [14387.361840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14387.372461] Lustre: Skipped 3 previous similar messages [14390.583726] Lustre: server umount lustre-MDT0000 complete [14391.457426] LustreError: 454243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14391.493761] LustreError: 454243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14394.200393] LustreError: 454228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788635681 with bad export cookie 4982658081543098595 [14394.207256] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14394.214547] LustreError: 454228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14394.531963] Lustre: server umount lustre-MDT0001 complete [14419.615168] LNet: 1 local NIs in recovery (showing 1): 192.168.204.129@tcp [14428.823680] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [14438.543390] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [14476.986161] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:16:02 (1788635762) [14496.648355] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [14500.274560] Key type lgssc unregistered [14500.553479] LNet: 460870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14500.561800] LNetError: 460870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14616.294161] LNet: Removed LNI 192.168.204.129@tcp [14616.977871] Key type .llcrypt unregistered [14616.980284] Key type ._llcrypt unregistered [14634.552795] Key type ._llcrypt registered [14634.554686] Key type .llcrypt registered [14634.809894] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [14635.989819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14636.025649] alg: No test for adler32 (adler32-zlib) [14637.261514] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [14637.611143] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [14639.335180] Key type lgssc registered [14640.468388] Lustre: Echo OBD driver; http://www.lustre.org/ [14644.581659] LNet: Added LNI 192.168.204.129@tcp100 [8/256/0/180] [14645.372144] LNet: Added LNI 192.168.204.129@tcp101 [8/256/0/180] [14646.475818] LNet: Added LNI 192.168.204.129@tcp102 [8/256/0/180] [14647.263963] LNet: Added LNI 192.168.204.129@tcp103 [8/256/0/180] [14648.240633] LNet: Added LNI 192.168.204.129@tcp104 [8/256/0/180] [14649.113249] LNet: Added LNI 192.168.204.129@tcp105 [8/256/0/180] [14649.997579] LNet: Added LNI 192.168.204.129@tcp106 [8/256/0/180] [14650.789563] LNet: Added LNI 192.168.204.129@tcp107 [8/256/0/180] [14651.584079] LNet: Added LNI 192.168.204.129@tcp108 [8/256/0/180] [14652.561202] LNet: Added LNI 192.168.204.129@tcp109 [8/256/0/180] [14653.347793] LNet: Added LNI 192.168.204.129@tcp110 [8/256/0/180] [14654.164227] LNet: Added LNI 192.168.204.129@tcp111 [8/256/0/180] [14654.943646] LNet: Added LNI 192.168.204.129@tcp112 [8/256/0/180] [14655.919310] LNet: Added LNI 192.168.204.129@tcp113 [8/256/0/180] [14656.846233] LNet: Added LNI 192.168.204.129@tcp114 [8/256/0/180] [14657.737960] LNet: Added LNI 192.168.204.129@tcp115 [8/256/0/180] [14658.705901] LNet: Added LNI 192.168.204.129@tcp116 [8/256/0/180] [14659.681416] LNet: Added LNI 192.168.204.129@tcp117 [8/256/0/180] [14660.712854] LNet: Added LNI 192.168.204.129@tcp118 [8/256/0/180] [14661.571731] LNet: Added LNI 192.168.204.129@tcp119 [8/256/0/180] [14662.478424] LNet: Added LNI 192.168.204.129@tcp120 [8/256/0/180] [14663.232190] LNet: Added LNI 192.168.204.129@tcp121 [8/256/0/180] [14664.260771] LNet: Added LNI 192.168.204.129@tcp122 [8/256/0/180] [14665.079367] LNet: Added LNI 192.168.204.129@tcp123 [8/256/0/180] [14666.057899] LNet: Added LNI 192.168.204.129@tcp124 [8/256/0/180] [14666.998400] LNet: Added LNI 192.168.204.129@tcp125 [8/256/0/180] [14668.106589] LNet: Added LNI 192.168.204.129@tcp126 [8/256/0/180] [14669.050943] LNet: Added LNI 192.168.204.129@tcp127 [8/256/0/180] [14670.047320] LNet: Added LNI 192.168.204.129@tcp128 [8/256/0/180] [14670.967932] LNet: Added LNI 192.168.204.129@tcp129 [8/256/0/180] [14671.803452] LNet: Added LNI 192.168.204.129@tcp130 [8/256/0/180] [14672.788805] LNet: Added LNI 192.168.204.129@tcp131 [8/256/0/180] [14673.747728] LNet: Added LNI 192.168.204.129@tcp132 [8/256/0/180] [14674.672153] LNet: Added LNI 192.168.204.129@tcp133 [8/256/0/180] [14675.885115] LNet: Added LNI 192.168.204.129@tcp134 [8/256/0/180] [14676.995925] LNet: Added LNI 192.168.204.129@tcp135 [8/256/0/180] [14684.258667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14684.317219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14685.512357] Lustre: 463902:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [14685.606983] LNetError: 461361:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.129 [14685.618987] LNetError: Protocol error connecting to 192.168.204.129@tcp121 at host 192.168.204.129:988: is it running a compatible version of Lustre? [14685.639144] LNetError: Skipped 9 previous similar messages [14685.758502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14685.791917] LNet: 6 local NIs in recovery (showing 6): 192.168.204.129@tcp125, 192.168.204.129@tcp126, 192.168.204.129@tcp127, 192.168.204.129@tcp128, 192.168.204.129@tcp129, 192.168.204.129@tcp121 [14685.795146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14685.938300] Lustre: lustre-MDT0000: new disk, initializing [14686.039604] Lustre: 463960:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [14686.053233] Lustre: 463960:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14686.077980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14686.107144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14686.694162] LNetError: 461361:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.129 [14686.705592] LNetError: 461361:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14686.714185] LNetError: Protocol error connecting to 192.168.204.129@tcp100 at host 192.168.204.129:988: is it running a compatible version of Lustre? [14688.865391] LNetError: 463923:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.129 [14688.879923] LNetError: 463923:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14688.891883] LNetError: Protocol error connecting to 192.168.204.129@tcp125 at host 192.168.204.129:988: is it running a compatible version of Lustre? [14688.911312] LNetError: Skipped 8 previous similar messages [14690.563187] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [14692.965470] LNetError: 461359:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.129 [14692.980474] LNetError: 461359:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14692.991888] LNetError: Protocol error connecting to 192.168.204.129@tcp125 at host 192.168.204.129:988: is it running a compatible version of Lustre? [14702.664062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14702.835071] Lustre: 463919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14702.955526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14702.975952] Lustre: Skipped 1 previous similar message [14703.131651] Lustre: lustre-MDT0001: new disk, initializing [14703.177807] Lustre: 464723:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [14703.196346] Lustre: 464723:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14703.312337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14703.361688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14703.379408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14708.552192] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [14712.438853] Lustre: 465110:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [14712.457467] Lustre: 465110:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14716.448561] LNetError: 463929:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.129 [14716.486581] LNetError: Protocol error connecting to 192.168.204.129@tcp122 at host 192.168.204.129:988: is it running a compatible version of Lustre? [14721.572939] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14724.641299] LNetError: 463937:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.129 [14724.658220] LNetError: 463937:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14724.678465] LNetError: Protocol error connecting to 192.168.204.129@tcp100 at host 192.168.204.129:988: is it running a compatible version of Lustre? [14724.690229] LNetError: Skipped 3 previous similar messages [14730.748678] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14739.968610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14740.454890] Lustre: lustre-OST0000: new disk, initializing [14740.460033] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14740.467721] Lustre: 465897:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14740.595595] Lustre: 465902:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.129@tcp: can't add NID 192.168.204.129@tcp130: rc = -75 [14740.615779] Lustre: 465902:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14740.736605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14741.096971] LNetError: 463937:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.129 [14741.113714] LNetError: 463937:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14741.127600] LNetError: Protocol error connecting to 192.168.204.129@tcp127 at host 192.168.204.129:988: is it running a compatible version of Lustre? [14741.164502] LNetError: Skipped 11 previous similar messages [14746.216097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14746.229627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14746.276186] LNet: 1 local NIs in recovery (showing 1): 192.168.204.129@tcp125 [14746.359466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14747.716915] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [14758.359980] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14767.076508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14767.088243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14767.099875] Lustre: Skipped 1 previous similar message [14772.197236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14772.203784] Lustre: Skipped 1 previous similar message [14772.682839] Lustre: server umount lustre-OST0000 complete [14775.968603] LNetError: 461359:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.129 [14775.985341] LNetError: 461359:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [14776.002632] LNetError: Protocol error connecting to 192.168.204.129@tcp100 at host 192.168.204.129:988: is it running a compatible version of Lustre? [14776.012352] LNetError: Skipped 16 previous similar messages [14777.321392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14777.333770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14777.341499] Lustre: Skipped 1 previous similar message [14777.360026] Lustre: Skipped 1 previous similar message [14782.433272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14782.450796] Lustre: Skipped 2 previous similar messages [14783.664887] Lustre: server umount lustre-MDT0000 complete [14788.811959] LustreError: 463910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636076 with bad export cookie 4221177254170884364 [14788.814356] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14788.845878] LustreError: 463910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14789.999295] Lustre: server umount lustre-MDT0001 complete [14801.558996] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [14804.770116] Key type lgssc unregistered [14805.122450] LNet: 467808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14805.132177] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14805.145389] LNet: Removed LNI 192.168.204.129@tcp [14806.175254] LNet: Removed LNI 192.168.204.129@tcp100 [14806.178091] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14806.186572] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14806.198139] LNet: Removed LNI 192.168.204.129@tcp101 [14807.264653] LNet: Removed LNI 192.168.204.129@tcp102 [14807.272590] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14807.293660] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14807.315098] LNet: Removed LNI 192.168.204.129@tcp103 [14808.351207] LNet: Removed LNI 192.168.204.129@tcp104 [14809.375232] LNet: Removed LNI 192.168.204.129@tcp105 [14809.377546] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14809.382662] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14809.389679] LNet: Removed LNI 192.168.204.129@tcp106 [14810.399543] LNet: Removed LNI 192.168.204.129@tcp107 [14811.424267] LNet: Removed LNI 192.168.204.129@tcp108 [14812.447567] LNet: Removed LNI 192.168.204.129@tcp109 [14813.473813] LNet: Removed LNI 192.168.204.129@tcp110 [14813.476106] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14813.483857] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14813.494670] LNet: Removed LNI 192.168.204.129@tcp111 [14814.559323] LNet: Removed LNI 192.168.204.129@tcp112 [14815.583246] LNet: Removed LNI 192.168.204.129@tcp113 [14816.607188] LNet: Removed LNI 192.168.204.129@tcp114 [14817.631177] LNet: Removed LNI 192.168.204.129@tcp115 [14818.655792] LNet: Removed LNI 192.168.204.129@tcp116 [14819.679268] LNet: Removed LNI 192.168.204.129@tcp117 [14820.705717] LNet: Removed LNI 192.168.204.129@tcp118 [14821.727437] LNet: Removed LNI 192.168.204.129@tcp119 [14821.729766] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14821.736113] LNetError: 467808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14821.743914] LNet: Removed LNI 192.168.204.129@tcp120 [14822.751450] LNet: Removed LNI 192.168.204.129@tcp121 [14823.780979] LNet: Removed LNI 192.168.204.129@tcp122 [14824.799238] LNet: Removed LNI 192.168.204.129@tcp123 [14825.823816] LNet: Removed LNI 192.168.204.129@tcp124 [14825.833507] LNet: Removed LNI 192.168.204.129@tcp125 [14826.911315] LNet: Removed LNI 192.168.204.129@tcp126 [14827.935194] LNet: Removed LNI 192.168.204.129@tcp127 [14828.959640] LNet: Removed LNI 192.168.204.129@tcp128 [14829.983241] LNet: Removed LNI 192.168.204.129@tcp129 [14829.990233] LNet: Removed LNI 192.168.204.129@tcp130 [14829.995178] LNet: Removed LNI 192.168.204.129@tcp131 [14829.999138] LNet: Removed LNI 192.168.204.129@tcp132 [14830.002283] LNet: Removed LNI 192.168.204.129@tcp133 [14830.006347] LNet: Removed LNI 192.168.204.129@tcp134 [14830.022903] LNet: Removed LNI 192.168.204.129@tcp135 [14831.031760] Key type .llcrypt unregistered [14831.041582] Key type ._llcrypt unregistered [14856.747297] Key type ._llcrypt registered [14856.753583] Key type .llcrypt registered [14856.939506] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [14872.942032] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [14874.002600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14874.151251] alg: No test for adler32 (adler32-zlib) [14875.143646] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [14875.415116] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [14877.103220] Key type lgssc registered [14878.241494] Lustre: Echo OBD driver; http://www.lustre.org/ [14930.419094] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:23:35 (1788636215) [14953.587054] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [14956.831202] Key type lgssc unregistered [14957.249583] LNet: 471898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14957.257873] LNetError: 471898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14957.273670] LNet: Removed LNI 192.168.204.129@tcp [14958.277591] Key type .llcrypt unregistered [14958.279353] Key type ._llcrypt unregistered [14978.193718] Key type ._llcrypt registered [14978.196388] Key type .llcrypt registered [14978.303571] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [14979.947356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14979.966578] alg: No test for adler32 (adler32-zlib) [14981.014957] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [14981.281126] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [14982.975285] Key type lgssc registered [14984.430465] Lustre: Echo OBD driver; http://www.lustre.org/ [14994.986863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14995.027454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14996.366560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14996.426196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14996.550934] Lustre: lustre-MDT0000: new disk, initializing [14996.720825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14996.769488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15001.865551] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15013.044296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15013.162613] Lustre: 473107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15013.207561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15013.212039] Lustre: Skipped 1 previous similar message [15013.305404] Lustre: lustre-MDT0001: new disk, initializing [15013.421339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15013.495076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15013.517583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15018.443560] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15028.969373] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15034.965407] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15042.455984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15042.768986] Lustre: lustre-OST0000: new disk, initializing [15042.779407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15042.789893] Lustre: 475077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15042.895676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15049.657447] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15050.801985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15050.814615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15050.935353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15059.883605] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15064.526577] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing lnet_if_list [15070.285323] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15070.340360] LNet: Added LNI 192.168.204.129@tcp10 [8/256/0/180] [15080.273140] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15080.335390] LNet: Added LNI 192.168.204.129@tcp11 [8/256/0/180] [15088.143022] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15088.248714] LNet: Added LNI 192.168.204.129@tcp12 [8/256/0/180] [15094.882159] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15094.903352] LNet: Added LNI 192.168.204.129@tcp13 [8/256/0/180] [15102.591514] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15102.683253] LNet: Added LNI 192.168.204.129@tcp14 [8/256/0/180] [15110.063286] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15110.090755] LNet: Added LNI 192.168.204.129@tcp15 [8/256/0/180] [15116.877137] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15116.927446] LNet: Added LNI 192.168.204.129@tcp16 [8/256/0/180] [15122.223522] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15122.278825] LNet: Added LNI 192.168.204.129@tcp17 [8/256/0/180] [15129.244588] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15129.325974] LNet: Added LNI 192.168.204.129@tcp18 [8/256/0/180] [15136.085639] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15136.118164] LNet: Added LNI 192.168.204.129@tcp19 [8/256/0/180] [15142.972933] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15143.036512] LNet: Added LNI 192.168.204.129@tcp20 [8/256/0/180] [15149.643956] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15149.672751] LNet: Added LNI 192.168.204.129@tcp21 [8/256/0/180] [15155.637084] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15155.692082] LNet: Added LNI 192.168.204.129@tcp22 [8/256/0/180] [15161.658462] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15161.702677] LNet: Added LNI 192.168.204.129@tcp23 [8/256/0/180] [15167.681913] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15167.717052] LNet: Added LNI 192.168.204.129@tcp24 [8/256/0/180] [15173.628115] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15173.677665] LNet: Added LNI 192.168.204.129@tcp25 [8/256/0/180] [15180.111304] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15180.145366] LNet: Added LNI 192.168.204.129@tcp26 [8/256/0/180] [15187.106744] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15187.144567] LNet: Added LNI 192.168.204.129@tcp27 [8/256/0/180] [15194.083987] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15194.125176] LNet: Added LNI 192.168.204.129@tcp28 [8/256/0/180] [15200.472453] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15200.499522] LNet: Added LNI 192.168.204.129@tcp29 [8/256/0/180] [15206.967586] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15207.011857] LNet: Added LNI 192.168.204.129@tcp30 [8/256/0/180] [15213.859925] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15213.896428] LNet: Added LNI 192.168.204.129@tcp31 [8/256/0/180] [15221.140397] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15221.180830] LNet: Added LNI 192.168.204.129@tcp32 [8/256/0/180] [15228.129387] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15228.171276] LNet: Added LNI 192.168.204.129@tcp33 [8/256/0/180] [15235.484566] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15235.514764] LNet: Added LNI 192.168.204.129@tcp34 [8/256/0/180] [15243.953386] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15244.005263] LNet: Added LNI 192.168.204.129@tcp35 [8/256/0/180] [15253.176277] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15253.207295] LNet: Added LNI 192.168.204.129@tcp36 [8/256/0/180] [15263.622737] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15263.668699] LNet: Added LNI 192.168.204.129@tcp37 [8/256/0/180] [15272.673149] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15272.721682] LNet: Added LNI 192.168.204.129@tcp38 [8/256/0/180] [15279.748969] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15279.777495] LNet: Added LNI 192.168.204.129@tcp39 [8/256/0/180] [15289.426251] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15289.457364] LNet: Added LNI 192.168.204.129@tcp40 [8/256/0/180] [15299.233972] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15299.269061] LNet: Added LNI 192.168.204.129@tcp41 [8/256/0/180] [15307.176746] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15307.204084] LNet: Added LNI 192.168.204.129@tcp42 [8/256/0/180] [15316.567992] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15316.609172] LNet: Added LNI 192.168.204.129@tcp43 [8/256/0/180] [15326.279206] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15326.334356] LNet: Added LNI 192.168.204.129@tcp44 [8/256/0/180] [15335.658991] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15335.705299] LNet: Added LNI 192.168.204.129@tcp45 [8/256/0/180] [15344.814589] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15344.855448] LNet: Added LNI 192.168.204.129@tcp46 [8/256/0/180] [15353.922522] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15353.950154] LNet: Added LNI 192.168.204.129@tcp47 [8/256/0/180] [15364.569068] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15364.615798] LNet: Added LNI 192.168.204.129@tcp48 [8/256/0/180] [15375.366331] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15375.419485] LNet: Added LNI 192.168.204.129@tcp49 [8/256/0/180] [15387.651149] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15387.677260] LNet: Added LNI 192.168.204.129@tcp50 [8/256/0/180] [15412.711173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15412.728622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15412.750698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15414.242099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15414.248669] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15414.270453] Lustre: Skipped 1 previous similar message [15416.952375] Lustre: server umount lustre-MDT0000 complete [15421.109594] LustreError: 475076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636708 with bad export cookie 3646597580549242114 [15421.111875] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15421.129084] LustreError: 475076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15421.561734] Lustre: server umount lustre-MDT0001 complete [15440.870646] Lustre: 472379:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788636712/real 1788636712] req@000000001a2aa072 x1875521063559040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788636728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15440.897469] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15442.911297] Lustre: 472381:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788636714/real 1788636714] req@00000000e5ea66ce x1875521063559168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788636730 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15442.991187] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [15446.056197] Lustre: 472381:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788636717/real 1788636717] req@000000001ddf0870 x1875521063559296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788636733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15452.128122] Lustre: 472380:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788636723/real 1788636723] req@0000000036be72a7 x1875521063559680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788636739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15452.196975] Lustre: 472380:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15453.070759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15457.249292] Lustre: 472381:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788636728/real 1788636728] req@00000000bc0aa041 x1875521063559808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788636744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15465.512329] LustreError: 472378:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008ee3e5aa x1875521063560448/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15465.943100] LustreError: 475070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15465.967944] LustreError: 475070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15466.050464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15470.064791] LustreError: 475071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15470.082715] LustreError: 475071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15471.081942] LustreError: 475070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15471.081963] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15471.190320] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15476.197138] LustreError: 475071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15481.316514] LustreError: 475070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15481.373312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15481.743027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15485.871377] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15487.200050] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15496.299897] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15502.706240] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15532.006713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15532.008642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15532.024351] Lustre: Skipped 1 previous similar message [15532.036367] Lustre: Skipped 3 previous similar messages [15535.013732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15535.029846] Lustre: Skipped 1 previous similar message [15537.449879] Lustre: server umount lustre-OST0000 complete [15543.267464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15543.277457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15543.304088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15548.357399] Lustre: server umount lustre-MDT0000 complete [15550.433822] LustreError: 499326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15550.447877] LustreError: 499326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15552.631365] LustreError: 475076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788636840 with bad export cookie 3646597580549244886 [15552.646500] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15552.876186] Lustre: server umount lustre-MDT0001 complete [15566.871455] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [15569.965719] Key type lgssc unregistered [15570.492843] LNet: 502125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15570.502423] LNetError: 502125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15570.518492] LNet: Removed LNI 192.168.204.129@tcp [15570.521380] LNet: Removed LNI 192.168.204.129@tcp10 [15570.524760] LNet: Removed LNI 192.168.204.129@tcp11 [15570.528480] LNet: Removed LNI 192.168.204.129@tcp12 [15570.535045] LNet: Removed LNI 192.168.204.129@tcp13 [15570.538824] LNet: Removed LNI 192.168.204.129@tcp14 [15570.542308] LNet: Removed LNI 192.168.204.129@tcp15 [15570.545083] LNet: Removed LNI 192.168.204.129@tcp16 [15570.548510] LNet: Removed LNI 192.168.204.129@tcp17 [15570.558920] LNet: Removed LNI 192.168.204.129@tcp18 [15570.562151] LNet: Removed LNI 192.168.204.129@tcp19 [15570.564435] LNet: Removed LNI 192.168.204.129@tcp20 [15570.566979] LNet: Removed LNI 192.168.204.129@tcp21 [15570.569540] LNet: Removed LNI 192.168.204.129@tcp22 [15570.573421] LNet: Removed LNI 192.168.204.129@tcp23 [15570.576409] LNet: Removed LNI 192.168.204.129@tcp24 [15570.578474] LNet: Removed LNI 192.168.204.129@tcp25 [15570.581432] LNet: Removed LNI 192.168.204.129@tcp26 [15570.584619] LNet: Removed LNI 192.168.204.129@tcp27 [15570.587437] LNet: Removed LNI 192.168.204.129@tcp28 [15570.590313] LNet: Removed LNI 192.168.204.129@tcp29 [15570.593523] LNet: Removed LNI 192.168.204.129@tcp30 [15570.596330] LNet: Removed LNI 192.168.204.129@tcp31 [15570.598887] LNet: Removed LNI 192.168.204.129@tcp32 [15570.601779] LNet: Removed LNI 192.168.204.129@tcp33 [15570.604291] LNet: Removed LNI 192.168.204.129@tcp34 [15570.607166] LNet: Removed LNI 192.168.204.129@tcp35 [15570.609272] LNet: Removed LNI 192.168.204.129@tcp36 [15570.612481] LNet: Removed LNI 192.168.204.129@tcp37 [15570.615346] LNet: Removed LNI 192.168.204.129@tcp38 [15570.617620] LNet: Removed LNI 192.168.204.129@tcp39 [15570.620364] LNet: Removed LNI 192.168.204.129@tcp40 [15570.622897] LNet: Removed LNI 192.168.204.129@tcp41 [15570.625315] LNet: Removed LNI 192.168.204.129@tcp42 [15570.628252] LNet: Removed LNI 192.168.204.129@tcp43 [15570.630814] LNet: Removed LNI 192.168.204.129@tcp44 [15570.633973] LNet: Removed LNI 192.168.204.129@tcp45 [15570.636732] LNet: Removed LNI 192.168.204.129@tcp46 [15570.639383] LNet: Removed LNI 192.168.204.129@tcp47 [15570.641926] LNet: Removed LNI 192.168.204.129@tcp48 [15570.644840] LNet: Removed LNI 192.168.204.129@tcp49 [15570.669581] LNet: Removed LNI 192.168.204.129@tcp50 [15571.590298] Key type .llcrypt unregistered [15571.593255] Key type ._llcrypt unregistered [15594.419619] Key type ._llcrypt registered [15594.425968] Key type .llcrypt registered [15594.555562] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [15612.360679] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [15613.324449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15613.353146] alg: No test for adler32 (adler32-zlib) [15614.607336] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [15614.952148] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [15616.831620] Key type lgssc registered [15618.601666] Lustre: Echo OBD driver; http://www.lustre.org/ [15664.448866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15664.493284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15666.023545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15666.053864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15666.185223] Lustre: lustre-MDT0000: new disk, initializing [15666.291084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15666.342268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15672.060940] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15684.928663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15685.070711] Lustre: 505324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15685.131686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15685.137143] Lustre: Skipped 1 previous similar message [15685.204685] Lustre: lustre-MDT0001: new disk, initializing [15685.419204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15685.468550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15685.490297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15690.872921] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15702.190434] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15709.353764] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15718.839883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15719.026867] Lustre: *** cfs_fail_loc=90f, val=0*** [15719.131499] Lustre: lustre-OST0000: new disk, initializing [15719.142730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15719.155651] Lustre: 507343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15719.258667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15725.158686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15725.181358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15725.322900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15727.587341] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15740.091224] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15751.141534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15751.143804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15751.164945] LustreError: Skipped 1 previous similar message [15751.168613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15751.190115] Lustre: Skipped 1 previous similar message [15751.219380] Lustre: Skipped 1 previous similar message [15756.049604] Lustre: server umount lustre-OST0000 complete [15761.381594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15761.405875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15766.657621] Lustre: server umount lustre-MDT0000 complete [15771.505643] LustreError: 505317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637059 with bad export cookie 4616393494820570451 [15771.518942] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15771.844462] Lustre: server umount lustre-MDT0001 complete [15785.744443] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [15789.893331] Key type lgssc unregistered [15790.254950] LNet: 509252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15790.268535] LNetError: 509252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15790.296992] LNet: Removed LNI 192.168.204.129@tcp [15791.584189] Key type .llcrypt unregistered [15791.586372] Key type ._llcrypt unregistered [15816.585265] Key type ._llcrypt registered [15816.587326] Key type .llcrypt registered [15816.711853] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [15818.596510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15818.629218] alg: No test for adler32 (adler32-zlib) [15819.968441] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [15820.333751] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [15822.071385] Key type lgssc registered [15823.553127] Lustre: Echo OBD driver; http://www.lustre.org/ [15833.873276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15833.913793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15835.591262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15840.897746] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15845.858763] LustreError: 510520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15845.878259] LustreError: 510520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15850.982134] LustreError: 510519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15851.231885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15851.716321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15856.357608] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15864.839809] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15871.876103] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15879.582713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15880.026264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15886.972429] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15895.502489] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15907.136731] loop0: detected capacity change from 0 to 204800000 [15907.334871] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15907.355087] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15907.392975] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15916.005482] loop0: detected capacity change from 0 to 204800000 [15916.293982] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15916.316838] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15916.344848] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15923.336463] loop0: detected capacity change from 0 to 204800000 [15929.962545] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15930.186662] Lustre: fs15246-OST0000: new disk, initializing [15930.190338] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15930.201803] Lustre: 514116:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15930.284783] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15938.135272] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15948.889545] loop1: detected capacity change from 0 to 204800000 [15955.496502] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15955.682121] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15955.767184] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15955.855396] Lustre: fs15246-MDT0000: new disk, initializing [15956.099950] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15956.170951] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15956.327256] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15957.413100] Lustre: 515242:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000cc2ab594 x1875521944210560/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788637261 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15961.096729] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [15962.482692] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15962.492936] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15962.550928] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15992.812604] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15992.830487] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15994.548277] Lustre: server umount fs15246-MDT0000 complete [15997.936406] LustreError: 511713:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16003.080571] LustreError: 511713:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16006.613195] Lustre: server umount fs15246-OST0000 complete [16023.522101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16023.534745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16023.541349] Lustre: Skipped 1 previous similar message [16023.555619] Lustre: Skipped 1 previous similar message [16028.644941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16028.658992] Lustre: Skipped 1 previous similar message [16033.761472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16033.773733] Lustre: Skipped 1 previous similar message [16034.271104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16034.412646] Lustre: server umount lustre-OST0000 complete [16038.885562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16038.906065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16038.916758] Lustre: Skipped 1 previous similar message [16044.587680] Lustre: server umount lustre-MDT0000 complete [16048.817703] LustreError: 511254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637336 with bad export cookie 6318213472172019115 [16048.828810] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16048.843102] LustreError: 511254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16049.280415] Lustre: server umount lustre-MDT0001 complete [16063.158937] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [16066.369595] Key type lgssc unregistered [16066.897250] LNet: 518343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16066.906977] LNetError: 518343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16066.931561] LNet: Removed LNI 192.168.204.129@tcp [16068.410525] Key type .llcrypt unregistered [16068.416842] Key type ._llcrypt unregistered [16093.037710] Key type ._llcrypt registered [16093.040903] Key type .llcrypt registered [16093.243328] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [16094.391639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16094.428896] alg: No test for adler32 (adler32-zlib) [16095.474385] Lustre: Lustre: Build Version: 2.17.58_2_gf69a9da [16095.685650] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [16097.383177] Key type lgssc registered [16099.257575] Lustre: Echo OBD driver; http://www.lustre.org/ [16143.529929] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [16154.670330] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [16198.462748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16198.494478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16199.741097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16199.779558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16199.854576] Lustre: lustre-MDT0000: new disk, initializing [16199.909591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16199.931340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16204.290933] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [16216.001514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16216.164281] Lustre: 522809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16216.218284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16216.222136] Lustre: Skipped 1 previous similar message [16216.320766] Lustre: lustre-MDT0001: new disk, initializing [16216.476247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16216.518878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16216.533386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16221.135681] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [16230.776647] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16236.955045] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16243.859899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16244.171521] Lustre: lustre-OST0000: new disk, initializing [16244.179626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16244.187730] Lustre: 524779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16244.288427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16249.417000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16249.432155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16249.556851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16251.579623] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [16261.593506] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16266.983655] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16267.226957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16272.146954] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16272.389198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16275.431919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16275.437476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16275.452441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16275.458653] Lustre: Skipped 1 previous similar message [16280.685712] Lustre: server umount lustre-OST0000 complete [16285.675795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16285.687070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16285.691647] Lustre: Skipped 2 previous similar messages [16290.785893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16290.800143] Lustre: Skipped 1 previous similar message [16291.529817] Lustre: server umount lustre-MDT0000 complete [16295.275942] LustreError: 522801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788637582 with bad export cookie 10531928666484445738 [16295.283042] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16295.299759] LustreError: 522801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16295.703172] Lustre: server umount lustre-MDT0001 complete [16325.600956] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [16338.755939] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [16376.771983] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16118 sec ========= 15:47:41 (1788637661) [16378.634722] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:47:43 (1788637663) === [16383.058989] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:47:47 (1788637667) === [16423.271695] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [16426.104113] Key type lgssc unregistered [16426.455822] LNet: 531427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16426.465242] LNetError: 531427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16426.479643] LNet: Removed LNI 192.168.204.129@tcp [16427.344297] Key type .llcrypt unregistered [16427.349190] Key type ._llcrypt unregistered