[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495664991 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002404] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009171] LSM: Security Framework initializing [ 0.010067] Yama: becoming mindful. [ 0.011059] SELinux: Initializing. [ 0.012088] *** VALIDATE selinux *** [ 0.020488] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025916] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026191] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029143] *** VALIDATE tmpfs *** [ 0.031400] *** VALIDATE proc *** [ 0.032298] *** VALIDATE cgroup *** [ 0.033014] *** VALIDATE cgroup2 *** [ 0.034324] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035187] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037040] Spectre V2 : User space: Vulnerable [ 0.038014] Speculative Store Bypass: Vulnerable [ 0.041782] debug: unmapping init [mem 0xffffffff8aa59000-0xffffffff8aa60fff] [ 0.043821] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044758] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045033] ... version: 2 [ 0.046020] ... bit width: 48 [ 0.047016] ... generic registers: 4 [ 0.048017] ... value mask: 0000ffffffffffff [ 0.049020] ... max period: 00007fffffffffff [ 0.050019] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052342] rcu: Hierarchical SRCU implementation. [ 0.054801] smp: Bringing up secondary CPUs ... [ 0.055628] x86: Booting SMP configuration: [ 0.056038] .... node #0, CPUs: #1 #2 #3 [ 0.062435] smp: Brought up 1 node, 4 CPUs [ 0.064039] smpboot: Max logical packages: 1 [ 0.065017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128698] node 0 deferred pages initialised in 61ms [ 0.131136] devtmpfs: initialized [ 0.132278] x86/mm: Memory block size: 128MB [ 0.135953] gcov: version magic: 0x41383552 [ 0.138412] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141344] pinctrl core: initialized pinctrl subsystem [ 0.142267] [ 0.142987] ************************************************************* [ 0.143020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144084] ** ** [ 0.145021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146018] ** ** [ 0.147024] ** This means that this kernel is built to expose internal ** [ 0.148017] ** IOMMU data structures, which may compromise security on ** [ 0.149017] ** your system. ** [ 0.150018] ** ** [ 0.151079] ** If you see this message and you are not debugging the ** [ 0.152076] ** kernel, report this immediately to your vendor! ** [ 0.153014] ** ** [ 0.154022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155030] ************************************************************* [ 0.156862] NET: Registered protocol family 16 [ 0.157588] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.158168] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161036] cpuidle: using governor menu [ 0.162865] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.166724] PCI: Using configuration type 1 for base access [ 0.168194] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182204] cryptd: max_cpu_qlen set to 1000 [ 0.185276] ACPI: Added _OSI(Module Device) [ 0.186019] ACPI: Added _OSI(Processor Device) [ 0.187017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.188015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192040] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.195589] ACPI: Interpreter enabled [ 0.196089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.197019] ACPI: Using IOAPIC for interrupt routing [ 0.198136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.199395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.209810] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.210058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.211021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.212083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.214483] acpiphp: Slot [2] registered [ 0.215127] acpiphp: Slot [5] registered [ 0.216113] acpiphp: Slot [6] registered [ 0.217109] acpiphp: Slot [7] registered [ 0.218097] acpiphp: Slot [8] registered [ 0.219111] acpiphp: Slot [9] registered [ 0.220111] acpiphp: Slot [10] registered [ 0.221144] acpiphp: Slot [3] registered [ 0.222110] acpiphp: Slot [4] registered [ 0.223091] acpiphp: Slot [11] registered [ 0.224111] acpiphp: Slot [12] registered [ 0.225091] acpiphp: Slot [13] registered [ 0.226118] acpiphp: Slot [14] registered [ 0.227103] acpiphp: Slot [15] registered [ 0.228142] acpiphp: Slot [16] registered [ 0.229124] acpiphp: Slot [17] registered [ 0.230096] acpiphp: Slot [18] registered [ 0.231103] acpiphp: Slot [19] registered [ 0.232089] acpiphp: Slot [20] registered [ 0.233094] acpiphp: Slot [21] registered [ 0.234118] acpiphp: Slot [22] registered [ 0.235126] acpiphp: Slot [23] registered [ 0.236120] acpiphp: Slot [24] registered [ 0.237110] acpiphp: Slot [25] registered [ 0.238091] acpiphp: Slot [26] registered [ 0.239128] acpiphp: Slot [27] registered [ 0.240146] acpiphp: Slot [28] registered [ 0.241208] acpiphp: Slot [29] registered [ 0.242137] acpiphp: Slot [30] registered [ 0.244151] acpiphp: Slot [31] registered [ 0.246160] PCI host bridge to bus 0000:00 [ 0.248031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.250036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.252046] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.255032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.258038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.262039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.264215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.267081] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.270354] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.280787] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.286011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.291028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.293019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.297027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.299695] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.301951] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.305101] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.309765] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.315019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.329020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.336889] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.343512] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.358021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.370018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.394019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.406502] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.419028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.460000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.470023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.479021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.505021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.525229] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.538019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.548022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.575019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590340] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.606020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.619019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.664017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.674065] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.696019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.719023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.763020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.800328] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.802563] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.805534] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.808442] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.811257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.816168] iommu: Default domain type: Passthrough [ 0.818678] SCSI subsystem initialized [ 0.820155] ACPI: bus type USB registered [ 0.822175] usbcore: registered new interface driver usbfs [ 0.825666] usbcore: registered new interface driver hub [ 0.829188] usbcore: registered new device driver usb [ 0.833308] pps_core: LinuxPPS API ver. 1 registered [ 0.836016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.839083] PTP clock support registered [ 0.843048] EDAC MC: Ver: 3.0.0 [ 0.847027] PCI: Using ACPI for IRQ routing [ 0.848567] NetLabel: Initializing [ 0.851014] NetLabel: domain hash size = 128 [ 0.853016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.856124] NetLabel: unlabeled traffic allowed by default [ 0.859113] vgaarb: loaded [ 0.861316] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.863028] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.874656] clocksource: Switched to clocksource kvm-clock [ 1.010042] VFS: Disk quotas dquot_6.6.0 [ 1.011587] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.014337] *** VALIDATE ramfs *** [ 1.015622] *** VALIDATE hugetlbfs *** [ 1.017269] pnp: PnP ACPI init [ 1.019452] pnp: PnP ACPI: found 6 devices [ 1.038314] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.041604] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.043807] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.046159] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.048748] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.051278] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.054170] NET: Registered protocol family 2 [ 1.056657] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.061626] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.067176] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.075813] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.079747] TCP: Hash tables configured (established 65536 bind 65536) [ 1.084982] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.089461] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.095179] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.099504] NET: Registered protocol family 1 [ 1.102968] RPC: Registered named UNIX socket transport module. [ 1.105441] RPC: Registered udp transport module. [ 1.107406] RPC: Registered tcp transport module. [ 1.109324] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.112715] NET: Registered protocol family 44 [ 1.114601] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.116390] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.118549] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.121098] PCI: CLS 0 bytes, default 64 [ 1.122900] Unpacking initramfs... [ 2.738390] debug: unmapping init [mem 0xffff9801bcc54000-0xffff9801bffbffff] [ 2.742700] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.745165] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.748092] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.328375] Initialise system trusted keyrings [ 3.330367] Key type blacklist registered [ 3.332930] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.343952] zbud: loaded [ 3.347405] *** VALIDATE nfs *** [ 3.348910] *** VALIDATE nfs4 *** [ 3.351155] pstore: using deflate compression [ 3.356147] Platform Keyring initialized [ 3.517533] NET: Registered protocol family 38 [ 3.519748] Key type asymmetric registered [ 3.522247] Asymmetric key parser 'x509' registered [ 3.525125] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.531295] io scheduler mq-deadline registered [ 3.533295] io scheduler kyber registered [ 3.535726] io scheduler bfq registered [ 3.538470] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.543243] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.547600] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.551840] ACPI: Power Button [PWRF] [ 3.559398] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.569419] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.597547] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.612527] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.648765] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.687246] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.720977] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.725985] Non-volatile memory driver v1.3 [ 3.728398] Linux agpgart interface v0.103 [ 3.767230] virtio_blk virtio1: [vda] 146208 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.770502] vda: detected capacity change from 0 to 74858496 [ 3.795507] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.799102] vdb: detected capacity change from 0 to 1073741824 [ 3.827092] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.830328] vdc: detected capacity change from 0 to 2621440000 [ 3.856395] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.861255] vdd: detected capacity change from 0 to 2621440000 [ 3.884039] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.887946] vde: detected capacity change from 0 to 4294967296 [ 3.907171] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.911025] vdf: detected capacity change from 0 to 4294967296 [ 3.927055] libphy: Fixed MDIO Bus: probed [ 3.950001] usbcore: registered new interface driver usbserial_generic [ 3.953593] usbserial: USB Serial support registered for generic [ 3.956461] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.960756] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.962764] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.965672] mousedev: PS/2 mouse device common for all mice [ 3.968308] rtc_cmos 00:05: RTC can wake from S4 [ 3.972665] rtc_cmos 00:05: registered as rtc0 [ 3.974094] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.976734] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.981536] intel_pstate: CPU model not supported [ 3.989600] hid: raw HID events driver (C) Jiri Kosina [ 3.991946] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.995415] usbcore: registered new interface driver usbhid [ 4.002144] usbhid: USB HID core driver [ 4.003974] drop_monitor: Initializing network drop monitor service [ 4.006506] Initializing XFRM netlink socket [ 4.008884] NET: Registered protocol family 10 [ 4.012341] Segment Routing with IPv6 [ 4.014205] NET: Registered protocol family 17 [ 4.016487] mpls_gso: MPLS GSO support [ 4.020151] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.029198] RAS: Correctable Errors collector initialized. [ 4.041269] AVX version of gcm_enc/dec engaged. [ 4.043481] AES CTR mode by8 optimization enabled [ 4.161230] sched_clock: Marking stable (4161068828, 0)->(5315136276, -1154067448) [ 4.165669] registered taskstats version 1 [ 4.167919] Loading compiled-in X.509 certificates [ 4.171367] zswap: loaded using pool lzo/zbud [ 4.205139] Key type big_key registered [ 4.223188] Key type encrypted registered [ 4.225207] ima: No TPM chip found, activating TPM-bypass! [ 4.227245] ima: Allocated hash algorithm: sha1 [ 4.229163] ima: No architecture policies found [ 4.231135] evm: Initialising EVM extended attributes: [ 4.234178] evm: security.selinux [ 4.235523] evm: security.ima [ 4.236775] evm: security.capability [ 4.238258] evm: HMAC attrs: 0x1 [ 4.240904] rtc_cmos 00:05: setting system clock to 2026-08-24 05:13:56 UTC (1787548436) [ 4.248964] debug: unmapping init [mem 0xffffffff8ba03000-0xffffffff8bbfffff] [ 4.252276] debug: unmapping init [mem 0xffffffff8a782000-0xffffffff8aa58fff] [ 4.264237] Write protecting the kernel read-only data: 28672k [ 4.269736] debug: unmapping init [mem 0xffffffff88e03000-0xffffffff88ffffff] [ 4.273236] debug: unmapping init [mem 0xffffffff89714000-0xffffffff897fffff] [ 4.318968] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.328329] systemd[1]: Detected virtualization kvm. [ 4.330404] systemd[1]: Detected architecture x86-64. [ 4.332805] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.362128] systemd[1]: No hostname configured. [ 4.363936] systemd[1]: Set hostname to . [ 4.367196] random: systemd: uninitialized urandom read (16 bytes read) [ 4.370579] systemd[1]: Initializing machine ID from random generator. [ 4.450065] random: ln: uninitialized urandom read (6 bytes read) [ 4.648351] random: systemd: uninitialized urandom read (16 bytes read) [ 4.651405] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.665594] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.673825] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.248242] device-mapper: uevent: version 1.0.3 [ 6.250552] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] [ 7.332481] random: fast init done Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.418431] virtio_net virtio0 ens2: renamed from eth0 [ 7.972207] scsi host0: ata_piix [ 8.066280] scsi host1: ata_piix [ 8.076742] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.097382] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.764655] random: crng init done [ 12.767423] random: 7 urandom warning(s) missed due to ratelimiting [ 16.149909] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.513922] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.866882] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.195749] SELinux: Disabled at runtime. [ 26.345608] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.377139] systemd[1]: Detected virtualization kvm. [ 26.382555] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.102215] hrtimer: interrupt took 3582172 ns [ 30.125984] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.144516] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.183593] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.201462] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.211468] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.270484] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.328267] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ 30.626140] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 31.432224] systemd[1]: sysroot-var-lib-nfs-rpc_pipefs.mount: Succeeded. [ 31.462784] systemd[1]: sysroot.mount: Succeeded. [ 31.518207] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 33.493644] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.897482] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 36.713148] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.079887] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.276789] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [*** ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit)[ 46.780438] Key type dns_resolver registered [* ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root [ 47.589090] NFS: Registering the id_resolver key type [ 47.595041] Key type id_resolver registered support (17s / n[ 47.615762] Key type id_legacy registered o limit) [*** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg652-server login: [ 144.661445] libcfs: loading out-of-tree module taints kernel. [ 144.759875] Key type ._llcrypt registered [ 144.765925] Key type .llcrypt registered [ 144.891887] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 161.424561] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 163.010465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 163.023435] alg: No test for adler32 (adler32-zlib) [ 164.585545] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 165.384665] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 167.240169] Key type lgssc registered [ 169.309860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 185.521492] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 223.311091] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 235.219169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 235.263097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.538606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 236.605316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 236.746648] Lustre: lustre-MDT0000: new disk, initializing [ 236.834723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.849214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 242.170794] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.026280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 255.153551] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 255.178485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 255.182409] Lustre: Skipped 1 previous similar message [ 255.242325] Lustre: lustre-MDT0001: new disk, initializing [ 255.295925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.331048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 255.347199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 260.127544] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.218465] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 274.438391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.695387] Lustre: lustre-OST0000: new disk, initializing [ 274.698605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 274.702860] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 274.748299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 280.078751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 280.088845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 280.198389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 280.967694] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 294.828184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 294.955395] Lustre: lustre-OST0001: new disk, initializing [ 294.962060] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 294.967527] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 295.053959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.985992] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 304.145558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 304.154521] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 304.215132] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 312.883993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 317.365873] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 323.969079] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing check_logdir /tmp/testlogs/ [ 328.460352] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing yml_node [ 332.330143] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 334.855879] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 337.540935] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 339.769942] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 01:19:30 EDT 2026 [ 358.414417] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 359.921283] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 375.777405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.784067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.796166] Lustre: Skipped 3 previous similar messages [ 375.808906] Lustre: Skipped 3 previous similar messages [ 380.898363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.905485] Lustre: Skipped 3 previous similar messages [ 381.911126] Lustre: server umount lustre-MDT0000 complete [ 385.653538] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548817 with bad export cookie 9031166431994024508 [ 385.656087] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 385.662286] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 386.018101] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.034329] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.049102] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.055346] Lustre: Skipped 2 previous similar messages [ 392.154233] Lustre: server umount lustre-MDT0001 complete [ 396.077463] Lustre: server umount lustre-OST0000 complete [ 400.185745] Lustre: server umount lustre-OST0001 complete [ 407.497761] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 417.658631] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 458.195492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.615681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 458.642956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 458.704300] Lustre: lustre-MDT0000: new disk, initializing [ 458.811886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.840139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 464.305830] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 476.404055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.561775] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 476.619905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 476.624695] Lustre: Skipped 1 previous similar message [ 476.707471] Lustre: lustre-MDT0001: new disk, initializing [ 476.864087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 476.946641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 476.961643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 481.946940] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 493.684727] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 499.907694] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 507.072327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 507.387548] Lustre: lustre-OST0000: new disk, initializing [ 507.391353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 507.400780] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 507.510477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.618148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 508.648535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 508.847732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 514.347210] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 524.243967] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 528.926651] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 529.132979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 533.700891] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 533.933283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 539.627299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.630855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.636098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.650475] Lustre: Skipped 2 previous similar messages [ 539.666253] LustreError: Skipped 1 previous similar message [ 549.856908] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 549.878178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.891971] Lustre: Skipped 4 previous similar messages [ 550.201310] Lustre: server umount lustre-OST0000 complete [ 554.978782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.000531] Lustre: Skipped 1 previous similar message [ 560.489531] Lustre: server umount lustre-MDT0000 complete [ 564.506544] LustreError: 17055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548996 with bad export cookie 9031166431994026041 [ 564.512226] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.517448] LustreError: 17055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 564.902440] Lustre: server umount lustre-MDT0001 complete [ 569.376392] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:23:20 (1787549000) [ 571.879505] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 573.598802] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:23:25 (1787549005) [ 580.772902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.286423] LustreError: 19468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.311710] LustreError: 19468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 581.391262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.725963] LustreError: 19469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.791554] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 590.818945] LustreError: 19468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.836919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.938219] LustreError: 19469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.918711] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 610.105826] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 616.628713] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 623.951752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 631.237915] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 638.317065] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 652.268044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.272064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 652.290276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 652.294202] LustreError: Skipped 1 previous similar message [ 652.317365] Lustre: Skipped 5 previous similar messages [ 654.516902] Lustre: server umount lustre-OST0000 complete [ 662.508631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.534113] Lustre: Skipped 2 previous similar messages [ 665.113269] Lustre: server umount lustre-MDT0000 complete [ 667.617265] LustreError: 20411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.643471] LustreError: 20411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 668.748503] LustreError: 19450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549101 with bad export cookie 9031166431994026923 [ 668.777612] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.182927] Lustre: server umount lustre-MDT0001 complete [ 681.398900] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 685.173278] Key type lgssc unregistered [ 685.602831] LNet: 23004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.616827] LNetError: 23004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.645199] LNet: Removed LNI 192.168.206.152@tcp [ 686.925172] Key type .llcrypt unregistered [ 686.926936] Key type ._llcrypt unregistered [ 712.835329] Key type ._llcrypt registered [ 712.837228] Key type .llcrypt registered [ 713.000714] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 714.093818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 714.161835] alg: No test for adler32 (adler32-zlib) [ 715.243733] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 715.540816] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 717.176206] Key type lgssc registered [ 718.447744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 727.172634] loop: module loaded [ 727.259451] loop0: detected capacity change from 0 to 204800000 [ 727.522557] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 727.540943] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 727.934884] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.686752] loop0: detected capacity change from 0 to 204800000 [ 733.933191] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.958606] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.989519] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.967214] loop0: detected capacity change from 0 to 204800000 [ 745.147944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 745.219378] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.470613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 746.529653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 746.581970] Lustre: lustre-MDT0000: new disk, initializing [ 746.657345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.691299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 751.698134] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 760.661169] loop1: detected capacity change from 0 to 204800000 [ 767.260697] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 767.672796] Lustre: lustre-OST07c6: new disk, initializing [ 767.681773] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 767.692978] Lustre: Skipped 1 previous similar message [ 767.700682] Lustre: 26194:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 767.818440] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 775.412439] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 775.864891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 775.883288] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 775.947050] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 782.513192] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 782.534042] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 782.546709] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 782.560018] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 782.573777] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 782.584224] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 797.823666] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 797.833341] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 797.848200] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.858814] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 797.867968] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 797.876492] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 803.013405] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 803.027811] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 803.034464] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 803.044921] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.053538] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 803.069791] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.079965] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 803.095221] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.108986] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 803.116666] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.125042] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 803.135031] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 806.373563] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 806.398709] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 806.410206] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 806.427400] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 806.434799] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 806.441629] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 806.451514] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 806.460930] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 806.468812] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 806.475414] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 806.480475] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 806.493814] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.591246] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 811.605327] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 811.616114] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 811.625977] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.638103] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 811.645322] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.657087] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 811.665772] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.674850] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 811.690870] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.712667] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 811.720388] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 819.986923] Lustre: 26877:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 819.994566] Lustre: 26877:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 820.002637] Lustre: 26877:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 820.009501] Lustre: 26877:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 820.024868] Lustre: 26877:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 820.035113] Lustre: 26877:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 820.044754] Lustre: 26877:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 820.050093] Lustre: 26877:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 820.056385] Lustre: 26877:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 820.062018] Lustre: 26877:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 820.067055] Lustre: 26877:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 820.073408] Lustre: 26877:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 836.078926] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 836.088159] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 836.093996] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 836.100872] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 836.107245] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 836.113030] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 836.119447] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 836.126780] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 836.133565] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 836.146209] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 836.152184] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 836.156532] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 878.049237] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.067283] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 880.426590] Lustre: server umount lustre-OST07c6 complete [ 891.042849] Lustre: server umount lustre-MDT0000 complete [ 908.189352] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 911.214316] Key type lgssc unregistered [ 911.510458] LNet: 28755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.516264] LNetError: 28755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.529622] LNet: Removed LNI 192.168.206.152@tcp [ 912.570942] Key type .llcrypt unregistered [ 912.573879] Key type ._llcrypt unregistered [ 931.847486] Key type ._llcrypt registered [ 931.850699] Key type .llcrypt registered [ 931.966215] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 932.990369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.079513] alg: No test for adler32 (adler32-zlib) [ 934.170285] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 934.445177] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 936.120154] Key type lgssc registered [ 937.190712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 944.663731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 944.694720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.035972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.787368] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 956.392088] LustreError: 30027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.425566] LustreError: 30027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 959.335302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.787252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 963.852638] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 971.329378] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 976.742255] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 983.268364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.688159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.143962] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 990.905891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 995.852645] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1010.660309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1010.676291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.700148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1011.683581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1011.696550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1011.701101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.704740] LustreError: Skipped 1 previous similar message [ 1016.039698] Lustre: server umount lustre-OST0000 complete [ 1021.416747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1021.424913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.445082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1021.458029] Lustre: Skipped 1 previous similar message [ 1026.673499] Lustre: server umount lustre-MDT0000 complete [ 1027.044345] LustreError: 30028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.059248] LustreError: 30028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1030.375090] LustreError: 30006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549462 with bad export cookie 17694939719090290954 [ 1030.383977] LustreError: 30006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1030.385769] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.770800] Lustre: server umount lustre-MDT0001 complete [ 1042.652281] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1046.186803] Key type lgssc unregistered [ 1046.504730] LNet: 33769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.513942] LNetError: 33769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.591121] LNet: Removed LNI 192.168.206.152@tcp [ 1048.612104] Key type .llcrypt unregistered [ 1048.614946] Key type ._llcrypt unregistered [ 1068.770351] Key type ._llcrypt registered [ 1068.772542] Key type .llcrypt registered [ 1068.889725] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1069.595551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.676073] alg: No test for adler32 (adler32-zlib) [ 1070.643623] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 1070.833276] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1072.472170] Key type lgssc registered [ 1073.312708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1080.463121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1080.511877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.016565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.479750] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1092.580033] LustreError: 35038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.599213] LustreError: 35038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1092.686113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.104061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1096.882850] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1102.919792] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1107.940413] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1113.598992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1113.886666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1119.271549] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1126.059644] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1127.224877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1142.753470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.765758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.781900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.795045] Lustre: Skipped 1 previous similar message [ 1142.821461] Lustre: Skipped 1 previous similar message [ 1145.791593] Lustre: server umount lustre-OST0000 complete [ 1153.002132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.007094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.021324] Lustre: Skipped 1 previous similar message [ 1153.034702] Lustre: Skipped 1 previous similar message [ 1155.948216] Lustre: server umount lustre-MDT0000 complete [ 1158.115738] LustreError: 36328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.135890] LustreError: 36328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1160.085900] LustreError: 35019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549592 with bad export cookie 10597818336592911978 [ 1160.092767] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.097458] LustreError: 35019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1160.384531] Lustre: server umount lustre-MDT0001 complete [ 1171.868572] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1175.240102] Key type lgssc unregistered [ 1175.591543] LNet: 38594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.601783] LNetError: 38594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.679521] LNet: Removed LNI 192.168.206.152@tcp [ 1177.473070] Key type .llcrypt unregistered [ 1177.474840] Key type ._llcrypt unregistered [ 1202.255215] Key type ._llcrypt registered [ 1202.263533] Key type .llcrypt registered [ 1202.367608] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1203.540239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1203.572904] alg: No test for adler32 (adler32-zlib) [ 1204.592467] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 1204.801852] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1206.456251] Key type lgssc registered [ 1207.701711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1215.224128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1215.236580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.743295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.855929] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1226.216762] LustreError: 40010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.234770] LustreError: 40010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1229.280278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1229.529038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1233.851790] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1241.166417] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1247.387909] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1253.959668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1254.209457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.443583] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1261.381661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1267.628784] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1275.361371] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 1275.881031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1275.889907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.906778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1276.100617] Lustre: server umount lustre-MDT0000 complete [ 1276.900145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.903227] LustreError: 40006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.907971] Lustre: Skipped 1 previous similar message [ 1276.920507] LustreError: 40006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1279.491571] LustreError: 39989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549711 with bad export cookie 16177016155654466967 [ 1279.498251] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1279.503336] LustreError: 39989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1280.446969] LustreError: 42502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1280.465742] LustreError: 42502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1280.483869] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.52@tcp (stopping) [ 1282.018110] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.068577] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1284.068598] LustreError: 40004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.072309] Lustre: Skipped 1 previous similar message [ 1284.094380] LustreError: 40004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1286.100890] Lustre: server umount lustre-MDT0001 complete [ 1307.645821] Lustre: server umount lustre-OST0000 complete [ 1325.725549] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1328.566773] Key type lgssc unregistered [ 1328.877529] LNet: 43872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.891622] LNetError: 43872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.906327] LNet: Removed LNI 192.168.206.152@tcp [ 1329.757815] Key type .llcrypt unregistered [ 1329.767584] Key type ._llcrypt unregistered [ 1351.381563] Key type ._llcrypt registered [ 1351.384293] Key type .llcrypt registered [ 1351.468605] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1352.486546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1352.513659] alg: No test for adler32 (adler32-zlib) [ 1353.532405] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 1353.720241] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1355.369398] Key type lgssc registered [ 1356.478848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1362.993655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1363.014727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.503275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.260453] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1374.693161] LustreError: 45297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.711956] LustreError: 45297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1376.056858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.432539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1380.895829] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1387.114988] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1392.060518] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1398.347049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1398.763916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.644323] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1409.005754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1411.992971] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1419.233410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.241969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.246919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.252239] Lustre: Skipped 1 previous similar message [ 1419.270598] Lustre: Skipped 1 previous similar message [ 1419.749558] Lustre: lustre-OST0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 1424.354118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1424.364148] Lustre: Skipped 1 previous similar message [ 1424.799075] Lustre: server umount lustre-OST0000 complete [ 1444.840084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.849915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.861957] Lustre: Skipped 1 previous similar message [ 1444.881340] Lustre: Skipped 1 previous similar message [ 1449.000216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.007597] Lustre: Skipped 1 previous similar message [ 1449.673124] Lustre: server umount lustre-MDT0000 complete [ 1453.104340] LustreError: 45278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549885 with bad export cookie 10272034696868059272 [ 1453.108483] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1453.113327] LustreError: 45278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1459.788770] Lustre: server umount lustre-MDT0001 complete [ 1472.605585] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1476.508808] Key type lgssc unregistered [ 1476.886744] LNet: 49011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.902683] LNetError: 49011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.919493] LNet: Removed LNI 192.168.206.152@tcp [ 1478.179460] Key type .llcrypt unregistered [ 1478.182086] Key type ._llcrypt unregistered [ 1501.953323] Key type ._llcrypt registered [ 1501.961429] Key type .llcrypt registered [ 1502.107882] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1503.079822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1503.189348] alg: No test for adler32 (adler32-zlib) [ 1504.218884] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 1504.513987] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1506.224213] Key type lgssc registered [ 1507.382616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1514.100596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1514.113838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.501247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1519.546402] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1525.731469] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.743394] LustreError: 50428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1526.694339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.007258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1531.350877] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1538.227297] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1544.036370] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1550.242461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1550.685252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.855671] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1559.957189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1564.376398] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1570.353650] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1573.570440] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1585.905114] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-24 1h40m16s [ 1587.364907] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1590.756372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.760085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.776132] Lustre: Skipped 1 previous similar message [ 1590.795812] Lustre: Skipped 2 previous similar messages [ 1591.844698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.853706] Lustre: Skipped 2 previous similar messages [ 1593.855816] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 1593.873280] Lustre: Skipped 1 previous similar message [ 1595.736800] Lustre: server umount lustre-MDT0000 complete [ 1597.927969] LustreError: 50427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.945939] LustreError: 50427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1599.159832] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1604.089905] LustreError: 51155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1604.132952] LustreError: 51155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1606.245486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.424961] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1606.757311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.855182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1607.799033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1611.326415] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1615.139931] Lustre: DEBUG MARKER: Wait for df (36783) ... [ 1617.173196] Lustre: DEBUG MARKER: done [ 1623.551184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.561547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1623.595527] Lustre: Skipped 2 previous similar messages [ 1623.622727] Lustre: Skipped 1 previous similar message [ 1628.677689] Lustre: server umount lustre-OST0000 complete [ 1633.762720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.764112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.764942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1633.782421] Lustre: Skipped 1 previous similar message [ 1633.803331] Lustre: Skipped 1 previous similar message [ 1638.455567] Lustre: server umount lustre-MDT0000 complete [ 1640.931322] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.952511] LustreError: 50423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1641.854592] LustreError: 53056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550074 with bad export cookie 4651351669384070626 [ 1641.858724] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1641.863018] LustreError: 53056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1642.114740] Lustre: server umount lustre-MDT0001 complete [ 1651.704430] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1654.079591] Key type lgssc unregistered [ 1654.388636] LNet: 55354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.394994] LNetError: 55354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.413251] LNet: Removed LNI 192.168.206.152@tcp [ 1655.481407] Key type .llcrypt unregistered [ 1655.485307] Key type ._llcrypt unregistered [ 1690.252034] Key type ._llcrypt registered [ 1690.254569] Key type .llcrypt registered [ 1690.357296] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1691.350629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.369656] alg: No test for adler32 (adler32-zlib) [ 1692.397635] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 1692.633107] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1694.280211] Key type lgssc registered [ 1695.150534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1701.903978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1701.916679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1703.073555] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1703.086454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1703.189261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.813991] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1714.941797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.031165] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1715.060819] Lustre: 57564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1715.382541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1720.034633] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1728.197556] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1734.204793] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1741.079323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1741.382614] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1741.655434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.641259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1747.916516] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1755.540326] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1761.453356] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-24 1h43m13s [ 1762.963926] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1769.482820] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1771.583743] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1771.613459] LustreError: 59029:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1792.006444] Lustre: lustre-MDT0000: Client e19a5093-9ae9-43fd-a8d0-04a4b2c51681 (at 192.168.206.52@tcp) reconnecting [ 1811.704121] LustreError: 59029:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1814.057103] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1816.928705] Lustre: DEBUG MARKER: done [ 1824.233715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1824.245534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.275749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1824.750230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1824.767349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.809638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1829.741278] Lustre: server umount lustre-OST0000 complete [ 1834.070328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.087849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1834.092640] Lustre: Skipped 1 previous similar message [ 1834.114797] Lustre: Skipped 1 previous similar message [ 1840.445317] Lustre: server umount lustre-MDT0000 complete [ 1844.245353] LustreError: 57554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550276 with bad export cookie 9388577731459103217 [ 1844.259874] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1844.277561] LustreError: 57554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1844.790799] Lustre: server umount lustre-MDT0001 complete [ 1854.755346] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1857.924239] Key type lgssc unregistered [ 1858.220920] LNet: 61893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.227732] LNetError: 61893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.253198] LNet: Removed LNI 192.168.206.152@tcp [ 1859.345489] Key type .llcrypt unregistered [ 1859.349108] Key type ._llcrypt unregistered [ 1913.773195] Key type ._llcrypt registered [ 1913.776675] Key type .llcrypt registered [ 1913.933397] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1915.199320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1915.539089] alg: No test for adler32 (adler32-zlib) [ 1916.566791] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 1916.980644] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1918.680286] Key type lgssc registered [ 1919.876591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.985612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1923.995346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.283608] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1925.326371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1925.586399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.951382] Lustre: Failing over lustre-MDT0000 [ 1927.601845] Lustre: server umount lustre-MDT0000 complete [ 1935.054948] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:46:06 (1787550366) [ 1942.185888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.686451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.649156] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1956.874857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.990731] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1957.038296] Lustre: 64693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1957.312494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1962.656693] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1969.980775] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1976.020278] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1982.774684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.928056] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1983.164796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.047137] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1992.833259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1997.956448] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2004.869668] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2013.669720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2013.680708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.698280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2013.705903] Lustre: Skipped 1 previous similar message [ 2013.723283] Lustre: Skipped 2 previous similar messages [ 2018.123870] Lustre: server umount lustre-MDT0000 complete [ 2022.191667] LustreError: 64686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550454 with bad export cookie 11018382920524854304 [ 2022.199505] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.204675] LustreError: 64686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2022.502502] Lustre: server umount lustre-MDT0001 complete [ 2027.656645] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2032.942658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2040.293886] Lustre: 63618:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550456/real 1787550456] req@00000000acf56f11 x1874382394793600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2040.317737] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.333289] Lustre: Skipped 1 previous similar message [ 2041.843669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.128277] Lustre: 63617:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550460/real 1787550460] req@000000004bcafac3 x1874382394793728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550476 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2045.408280] Lustre: 63616:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550461/real 1787550461] req@000000008337364c x1874382394793856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2047.884719] LustreError: 66541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.893302] LustreError: 66541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2047.978422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.094590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2048.933703] Lustre: 63615:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550464/real 1787550464] req@000000005627b31e x1874382394794112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2050.019201] LustreError: 66542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.046397] LustreError: 66542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2052.869782] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2053.090458] Lustre: 63618:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550469/real 1787550469] req@00000000a084a7fa x1874382394794240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2055.139503] LustreError: 68464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.159591] LustreError: 68464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2060.261827] LustreError: 66541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.281174] LustreError: 66541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2061.924041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.283733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2067.316489] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2067.452751] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2069.544228] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2075.686736] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2080.997560] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2089.966792] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.969993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.992320] Lustre: Skipped 1 previous similar message [ 2090.005317] Lustre: Skipped 2 previous similar messages [ 2094.808495] Lustre: server umount lustre-MDT0000 complete [ 2095.077092] LustreError: 69907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.092271] LustreError: 69907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2098.276383] LustreError: 65437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550530 with bad export cookie 11018382920524856586 [ 2098.278473] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2098.282207] LustreError: 65437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2098.531802] Lustre: server umount lustre-MDT0001 complete [ 2105.211536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2113.931265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.320150] Lustre: 63616:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787550532/real 1787550532] req@0000000003908567 x1874382394822016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787550548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.339117] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.351675] Lustre: Skipped 1 previous similar message [ 2123.557156] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x98e928e3691d826d [ 2123.580463] Lustre: MGC192.168.206.152@tcp: Connection restored to 0@lo (at 0@lo) [ 2123.792407] LustreError: 69202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.850050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.904798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2127.954262] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2137.951751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.730942] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2144.466595] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2154.682382] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2163.087722] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2171.375990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.380951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2171.405471] Lustre: Skipped 2 previous similar messages [ 2171.425109] Lustre: Skipped 2 previous similar messages [ 2176.488576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.500979] Lustre: Skipped 2 previous similar messages [ 2177.261839] Lustre: server umount lustre-MDT0000 complete [ 2181.109076] LustreError: 64684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550613 with bad export cookie 11018382920524857965 [ 2181.120494] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2181.123941] LustreError: 64684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2181.490095] Lustre: server umount lustre-MDT0001 complete [ 2188.041216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2191.689506] Lustre: DEBUG MARKER: files compared the same [ 2204.002403] Lustre: server umount lustre-OST0000 complete [ 2222.569702] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 2226.368108] Key type lgssc unregistered [ 2226.817851] LNet: 74430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.834893] LNetError: 74430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.878833] LNet: Removed LNI 192.168.206.152@tcp [ 2227.982415] Key type .llcrypt unregistered [ 2227.984642] Key type ._llcrypt unregistered [ 2256.053906] Key type ._llcrypt registered [ 2256.060847] Key type .llcrypt registered [ 2256.265452] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2258.236934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2258.519057] alg: No test for adler32 (adler32-zlib) [ 2260.017662] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 2260.476602] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 2262.416416] Key type lgssc registered [ 2264.387549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2278.603472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2278.692141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.753766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.336867] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug +malloc all [ 2290.157215] LustreError: 75852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.179118] LustreError: 75852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2295.287818] LustreError: 75853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.860889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2296.200904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2301.425420] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug +malloc all [ 2311.407509] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.100900] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2325.204406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.562171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.713682] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug +malloc all [ 2341.093853] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2350.202846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2360.801147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.802815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.822721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.921870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.940355] Lustre: Skipped 1 previous similar message [ 2366.810630] Lustre: server umount lustre-OST0000 complete [ 2373.090772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2373.117576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.139130] Lustre: Skipped 1 previous similar message [ 2373.144240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2373.149727] Lustre: Skipped 1 previous similar message [ 2376.177410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.181573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.210773] Lustre: Skipped 1 previous similar message [ 2381.281819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2381.291141] Lustre: Skipped 1 previous similar message [ 2386.401757] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2386.689071] Lustre: server umount lustre-MDT0000 complete [ 2391.269779] LustreError: 75833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550823 with bad export cookie 10531840401608369531 [ 2391.272336] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2391.293651] LustreError: 75833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2391.739547] Lustre: server umount lustre-MDT0001 complete [ 2407.156221] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 2410.988430] Key type lgssc unregistered [ 2411.261373] LNet: 79413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.270639] LNetError: 79413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2412.327734] LNet: Removed LNI 192.168.206.152@tcp [ 2413.088976] Key type .llcrypt unregistered [ 2413.093049] Key type ._llcrypt unregistered [ 2441.340094] Key type ._llcrypt registered [ 2441.343317] Key type .llcrypt registered [ 2441.481627] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2443.059547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2443.103029] alg: No test for adler32 (adler32-zlib) [ 2444.299993] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 2444.649712] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 2446.408209] Key type lgssc registered [ 2447.884590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2457.253886] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2457.279527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2472.928528] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2478.048579] LustreError: 80804:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.152@tcp: failed processing log, type 4: rc = -110 [ 2504.675587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.790934] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2523.594232] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2533.494597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.684735] Lustre: *** cfs_fail_loc=706, val=0*** [ 2534.169631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.240792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2539.300044] LustreError: 80829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.315836] LustreError: 80829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2540.202220] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2544.615397] LustreError: 80830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.630343] LustreError: 80830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2549.731811] LustreError: 81890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.765720] LustreError: 81890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2551.701518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.303687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2558.874539] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2566.836818] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2574.769693] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2577.891763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.894975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2577.913828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.938520] Lustre: Skipped 1 previous similar message [ 2583.014660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2583.026287] Lustre: Skipped 1 previous similar message [ 2583.437940] Lustre: server umount lustre-OST0000 complete [ 2588.129200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2588.136718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.152680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.164610] Lustre: Skipped 2 previous similar messages [ 2588.176705] Lustre: Skipped 1 previous similar message [ 2593.250670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.261142] Lustre: Skipped 1 previous similar message [ 2593.472958] Lustre: server umount lustre-MDT0000 complete [ 2596.895512] LustreError: 80812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551029 with bad export cookie 17207818727246702499 [ 2596.898492] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2596.907815] LustreError: 80812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2597.219247] Lustre: server umount lustre-MDT0001 complete [ 2607.173337] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 2609.935505] Key type lgssc unregistered [ 2610.176727] LNet: 84426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2610.188048] LNetError: 84426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2610.198801] LNet: Removed LNI 192.168.206.152@tcp [ 2611.016726] Key type .llcrypt unregistered [ 2611.018560] Key type ._llcrypt unregistered [ 2636.584618] Key type ._llcrypt registered [ 2636.586530] Key type .llcrypt registered [ 2636.720349] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2637.567925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2637.628245] alg: No test for adler32 (adler32-zlib) [ 2639.142259] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 2639.483181] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 2641.217546] Key type lgssc registered [ 2643.694558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2654.911805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2654.934500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.563096] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2671.792773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.678064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2677.733409] LustreError: 86540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.763026] LustreError: 86540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2678.752259] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2681.828234] LustreError: 86539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.885906] LustreError: 86539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2686.945640] LustreError: 86544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.988128] LustreError: 86544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2689.068664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.680232] LustreError: 86539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.726810] LustreError: 86539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2689.747713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.151737] LustreError: 86538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.177453] LustreError: 86538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2697.118085] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2707.433410] LustreError: 87287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.449490] LustreError: 87287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2708.322915] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.330537] Lustre: MGS: Client 8121c06a-48c5-4a02-89a4-067135e1e24f (at 0@lo) reconnecting [ 2708.354867] LustreError: 86554:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2708.619116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.727672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2714.852507] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2728.930718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2728.935570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.944883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.446601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.461867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.465197] Lustre: Skipped 1 previous similar message [ 2733.367342] Lustre: server umount lustre-OST0000 complete [ 2739.683239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2739.685428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.695969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.721537] Lustre: Skipped 1 previous similar message [ 2739.743633] Lustre: Skipped 1 previous similar message [ 2743.129956] Lustre: server umount lustre-MDT0000 complete [ 2744.811925] LustreError: 88029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.847404] LustreError: 88029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2748.004519] Lustre: server umount lustre-MDT0001 complete [ 2751.630427] Lustre: server umount MGS complete [ 2762.832920] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:59:54 (1787551194) [ 2795.556467] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 2811.918589] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2852.889212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.413332] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2867.971212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.326379] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2868.394769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2868.504375] Lustre: lustre-MDT0001: new disk, initializing [ 2868.613538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2868.630689] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2868.636045] Lustre: 93745:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2868.642221] LustreError: 93745:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2873.099593] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2884.814415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2885.211803] Lustre: lustre-OST0000: new disk, initializing [ 2885.219716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2885.233149] Lustre: 94510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2885.353476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.868866] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2893.412532] Lustre: 94824:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008337364c x1874383151867648/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787551341 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2902.143806] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2909.855623] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.877851] Lustre: MGS: Client dcf542dc-1610-4c6a-9383-acd3ddf4a67f (at 0@lo) reconnecting [ 2909.969320] Lustre: 95556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2910.000272] LustreError: 93731:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2910.020125] LustreError: 93731:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2910.061354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2910.256422] Lustre: lustre-MDT0000: new disk, initializing [ 2910.499809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.562505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2910.607503] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2911.651757] Lustre: 95580:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004e76c139 x1874383151874560/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787551359 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2916.835917] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2916.866366] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2920.822302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2920.840462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2921.008846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2941.409832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2941.416261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.432259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2945.514673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2945.518660] Lustre: Skipped 2 previous similar messages [ 2955.232133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2955.452963] Lustre: server umount lustre-OST0000 complete [ 2960.866722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.867902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.880911] Lustre: Skipped 2 previous similar messages [ 2960.890884] Lustre: Skipped 3 previous similar messages [ 2973.152174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2973.275050] Lustre: server umount lustre-MDT0000 complete [ 2976.232699] LustreError: 93717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2976.248150] LustreError: 93717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2982.780946] Lustre: server umount lustre-MDT0001 complete [ 2985.992913] Lustre: server umount MGS complete [ 2996.978723] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:03:48 (1787551428) [ 3014.319224] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 3017.462907] Key type lgssc unregistered [ 3017.747667] LNet: 98088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3017.754361] LNetError: 98088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3017.773688] LNet: Removed LNI 192.168.206.152@tcp [ 3018.877853] Key type .llcrypt unregistered [ 3018.880091] Key type ._llcrypt unregistered [ 3044.999686] Key type ._llcrypt registered [ 3045.002902] Key type .llcrypt registered [ 3045.123500] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3047.019263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3047.102029] alg: No test for adler32 (adler32-zlib) [ 3048.246961] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 3048.463588] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 3050.248175] Key type lgssc registered [ 3052.374186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3062.220898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3062.239114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.980928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.809226] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3074.535497] LustreError: 99311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.571792] LustreError: 99311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3079.073777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.382832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3084.014611] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3090.736865] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3095.934277] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3102.988852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3103.307767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3109.330942] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3109.512557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3116.787491] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3125.217808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3125.225249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.238229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3125.238795] Lustre: Skipped 1 previous similar message [ 3125.248987] Lustre: Skipped 1 previous similar message [ 3130.345299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.352949] Lustre: Skipped 1 previous similar message [ 3131.135525] Lustre: server umount lustre-OST0000 complete [ 3135.457715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.494727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.517251] Lustre: Skipped 1 previous similar message [ 3140.577354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.589195] Lustre: Skipped 1 previous similar message [ 3142.967881] Lustre: server umount lustre-MDT0000 complete [ 3145.697383] LustreError: 99312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.474510] LustreError: 99293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551579 with bad export cookie 17182486651242471648 [ 3147.476633] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.482817] LustreError: 99293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3147.823991] Lustre: server umount lustre-MDT0001 complete [ 3161.229270] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 3165.836835] Key type lgssc unregistered [ 3166.126947] LNet: 102847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3166.150193] LNetError: 102847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3166.165368] LNet: Removed LNI 192.168.206.152@tcp [ 3167.696385] Key type .llcrypt unregistered [ 3167.702323] Key type ._llcrypt unregistered [ 3192.739353] Key type ._llcrypt registered [ 3192.742905] Key type .llcrypt registered [ 3192.885749] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3194.691765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3194.708904] alg: No test for adler32 (adler32-zlib) [ 3196.790792] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 3197.190127] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 3198.889209] Key type lgssc registered [ 3200.734632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3208.765644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3208.773348] LustreError: 103918:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3208.936520] LustreError: 103933:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3208.947033] LustreError: 103918:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4834 [ 3208.958161] LustreError: 103918:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3208.966160] LustreError: 103918:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3208.973449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.823974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.970712] LustreError: 103970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3220.993363] LustreError: 103970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3221.069285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.474092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.685472] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3233.662410] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing lsmod [ 3237.006592] LustreError: 105135:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3237.018333] LustreError: 105139:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3237.038104] LustreError: 105135:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3237.047355] LustreError: 105135:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3237.059390] LustreError: 105135:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3237.114051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.573629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3241.953167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3241.961682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.988753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3242.979227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3242.982161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3243.021344] Lustre: Skipped 1 previous similar message [ 3243.041455] Lustre: Skipped 2 previous similar messages [ 3246.124455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.145615] Lustre: Skipped 2 previous similar messages [ 3247.011143] Lustre: server umount lustre-MDT0000 complete [ 3251.172136] LustreError: 103971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.193436] LustreError: 103971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3251.839434] LustreError: 103949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551684 with bad export cookie 7079626226894732071 [ 3251.851334] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3251.862815] LustreError: 103949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3251.911656] Lustre: 103962:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.152@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3252.351373] LustreError: 104402:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000025948339 x1874383736950912/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3252.576961] Lustre: server umount lustre-MDT0001 complete [ 3266.958780] Lustre: server umount lustre-OST0000 complete [ 3276.249731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.849741] LustreError: 106203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.924995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.902800] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3282.414611] LustreError: 106204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.536630] LustreError: 106203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.023780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.513260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3295.913241] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3303.728794] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3310.138795] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3318.043956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3318.511312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3322.617930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3325.111531] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3334.042730] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3351.524067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.532973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3351.551054] Lustre: Skipped 1 previous similar message [ 3351.594768] Lustre: Skipped 1 previous similar message [ 3355.135229] Lustre: server umount lustre-OST0000 complete [ 3361.761494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.794362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3361.811898] Lustre: Skipped 1 previous similar message [ 3366.032415] Lustre: server umount lustre-MDT0000 complete [ 3366.887902] LustreError: 106203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.907291] LustreError: 106203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3369.990664] LustreError: 106186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551802 with bad export cookie 7079626226894732764 [ 3370.002519] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3370.010096] LustreError: 106186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3370.454299] Lustre: server umount lustre-MDT0001 complete [ 3383.577360] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 3387.223487] Key type lgssc unregistered [ 3387.584160] LNet: 109764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3387.609122] LNetError: 109764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3387.642753] LNet: Removed LNI 192.168.206.152@tcp [ 3388.724796] Key type .llcrypt unregistered [ 3388.728980] Key type ._llcrypt unregistered [ 3420.151929] Key type ._llcrypt registered [ 3420.159881] Key type .llcrypt registered [ 3420.303930] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3421.411835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3421.455204] alg: No test for adler32 (adler32-zlib) [ 3423.135679] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 3423.405220] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 3425.080160] Key type lgssc registered [ 3426.303960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3438.287606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3438.343924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.291796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.663684] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3449.827780] LustreError: 111184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.854573] LustreError: 111184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3454.948185] LustreError: 111185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.410638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.863540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3460.797480] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3468.499204] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3475.551990] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3482.824696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3483.177428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3485.292638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3490.417607] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3500.355828] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3505.873413] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3510.881611] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3520.999627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3521.012789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.021418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3523.092608] Lustre: server umount lustre-OST0000 complete [ 3531.240088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.268059] Lustre: Skipped 1 previous similar message [ 3531.277614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.292363] Lustre: Skipped 2 previous similar messages [ 3534.969421] Lustre: server umount lustre-MDT0000 complete [ 3536.353359] LustreError: 111181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.379026] LustreError: 111181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3540.039569] LustreError: 111165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551972 with bad export cookie 9235289530844659396 [ 3540.043018] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3540.056797] LustreError: 111165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3540.776989] Lustre: server umount lustre-MDT0001 complete [ 3555.585768] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 3559.485723] Key type lgssc unregistered [ 3559.918891] LNet: 114837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3559.924284] LNetError: 114837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3559.944367] LNet: Removed LNI 192.168.206.152@tcp [ 3560.904261] Key type .llcrypt unregistered [ 3560.907484] Key type ._llcrypt unregistered [ 3581.396281] Key type ._llcrypt registered [ 3581.406732] Key type .llcrypt registered [ 3581.595957] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3583.229489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3583.330782] alg: No test for adler32 (adler32-zlib) [ 3584.429744] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 3584.736917] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 3586.448162] Key type lgssc registered [ 3587.871534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3597.402472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3597.461150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.123750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.670969] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3609.577912] LustreError: 116059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.604901] LustreError: 116059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3614.067330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.535386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3619.711386] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3627.626517] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3634.187781] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3643.122812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3643.629049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.651499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3651.241153] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3661.078688] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3674.586515] Lustre: server umount lustre-OST0000 complete [ 3679.716966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.721239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3679.751378] Lustre: Skipped 1 previous similar message [ 3679.792826] Lustre: Skipped 1 previous similar message [ 3684.834660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3684.843996] Lustre: Skipped 1 previous similar message [ 3689.037894] Lustre: server umount lustre-MDT0000 complete [ 3689.952339] Lustre: 115371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787552106/real 1787552106] req@00000000749aa44b x1874384142758528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787552122 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3689.973136] LustreError: 117050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3689.979367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.997659] LustreError: 117050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3692.912967] LustreError: 116040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552125 with bad export cookie 13306029493407883911 [ 3692.930832] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3693.480074] Lustre: server umount lustre-MDT0001 complete [ 3705.947993] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 3709.211619] Key type lgssc unregistered [ 3709.502687] LNet: 119597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3709.517864] LNetError: 119597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3709.534270] LNet: Removed LNI 192.168.206.152@tcp [ 3710.634763] Key type .llcrypt unregistered [ 3710.638472] Key type ._llcrypt unregistered [ 3736.986865] Key type ._llcrypt registered [ 3736.989467] Key type .llcrypt registered [ 3737.168947] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3738.268051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3738.413057] alg: No test for adler32 (adler32-zlib) [ 3739.520640] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 3739.823711] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 3741.520714] Key type lgssc registered [ 3742.884778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3751.440203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3751.453704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.962964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.104483] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3763.184977] LustreError: 121017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3763.205277] LustreError: 121017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3767.008332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.395705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3771.896462] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3780.039904] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3786.032262] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3792.701778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3792.972557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.138220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3799.542839] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3807.355555] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3818.163818] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3821.973748] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3821.978257] Lustre: Skipped 2 previous similar messages [ 3826.497427] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3826.507864] Lustre: Skipped 2 previous similar messages [ 3828.371554] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3828.400679] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3833.944564] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3833.961791] Lustre: Skipped 2 previous similar messages [ 3834.484207] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.52@tcp 192.168.0.[2,10]@tcp [ 3838.949132] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.52/24@tcp in log lustre-MDT0000 [ 3838.958414] Lustre: Skipped 2 previous similar messages [ 3840.631618] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.52/24@tcp [ 3840.640119] Lustre: Skipped 1 previous similar message [ 3849.700407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3849.705775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.724376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.805163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3853.811840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3853.825316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.855164] Lustre: Skipped 1 previous similar message [ 3855.901507] Lustre: server umount lustre-OST0000 complete [ 3864.039489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.052133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.915061] Lustre: server umount lustre-MDT0000 complete [ 3869.155336] LustreError: 121018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3869.190605] LustreError: 121018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3871.466345] LustreError: 121751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552303 with bad export cookie 13727633740342899779 [ 3871.476368] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3871.488785] LustreError: 121751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3871.814832] Lustre: server umount lustre-MDT0001 complete [ 3885.675783] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 3889.474425] Key type lgssc unregistered [ 3890.040477] LNet: 125665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3890.058652] LNetError: 125665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3890.074692] LNet: Removed LNI 192.168.206.152@tcp [ 3891.171790] Key type .llcrypt unregistered [ 3891.179211] Key type ._llcrypt unregistered [ 3928.400897] Key type ._llcrypt registered [ 3928.403212] Key type .llcrypt registered [ 3928.477968] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3929.996391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3930.019521] alg: No test for adler32 (adler32-zlib) [ 3931.068954] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 3931.346556] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 3933.034809] Key type lgssc registered [ 3934.402534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3942.542604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3942.562438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.024833] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3944.052594] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.52/24@tcp [ 3944.057764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.359250] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3954.661970] LustreError: 127230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3954.680510] LustreError: 127230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3956.764572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.028808] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3957.053771] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.52/24@tcp [ 3957.105369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3961.023526] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3967.193337] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3972.561115] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3980.033427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3980.671878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.569664] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3988.963875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3995.288349] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4008.421391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4008.434661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.455302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.443814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.446490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.449831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4011.434865] Lustre: server umount lustre-OST0000 complete [ 4018.657681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4018.678430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.699569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.709019] Lustre: Skipped 1 previous similar message [ 4022.257980] Lustre: server umount lustre-MDT0000 complete [ 4024.825526] LustreError: 127230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.866462] LustreError: 127230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4027.183617] LustreError: 127212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552459 with bad export cookie 5933120340984843435 [ 4027.186351] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4027.194681] LustreError: 127212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4028.222541] Lustre: server umount lustre-MDT0001 complete [ 4039.736917] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4043.198988] Key type lgssc unregistered [ 4043.676863] LNet: 130822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4043.681422] LNetError: 130822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4043.696526] LNet: Removed LNI 192.168.206.152@tcp [ 4044.908889] Key type .llcrypt unregistered [ 4044.911412] Key type ._llcrypt unregistered [ 4069.741111] Key type ._llcrypt registered [ 4069.745833] Key type .llcrypt registered [ 4069.865432] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 4084.000321] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4085.107881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4085.239320] alg: No test for adler32 (adler32-zlib) [ 4086.250642] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 4086.513258] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4088.208219] Key type lgssc registered [ 4089.126170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4120.031968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4120.056436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.242561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4121.275283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4121.345553] Lustre: lustre-MDT0000: new disk, initializing [ 4121.518166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4121.536783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4125.904838] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4137.254786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.450813] Lustre: 134221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4137.540616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4137.549133] Lustre: Skipped 1 previous similar message [ 4137.707673] Lustre: lustre-MDT0001: new disk, initializing [ 4137.895410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4137.929688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4137.943554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4142.480728] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4152.773048] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4158.760329] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4165.343518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4165.511467] Lustre: lustre-OST0000: new disk, initializing [ 4165.515259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4165.520880] Lustre: 136191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4165.591762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.488223] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4173.875585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4173.889150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4173.960154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4181.671855] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4187.483504] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4187.755404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4194.222709] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4194.500334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4199.908622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4199.920172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.934791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4199.947285] Lustre: Skipped 1 previous similar message [ 4202.664727] Lustre: server umount lustre-OST0000 complete [ 4209.633456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4209.648774] LustreError: Skipped 1 previous similar message [ 4209.657520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4209.687220] Lustre: Skipped 1 previous similar message [ 4209.706235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4212.991880] Lustre: server umount lustre-MDT0000 complete [ 4217.011485] LustreError: 134213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552649 with bad export cookie 18297575969740951862 [ 4217.021241] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4217.026686] LustreError: 134213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4217.460635] Lustre: server umount lustre-MDT0001 complete [ 4227.444188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.890643] LustreError: 138408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.910571] LustreError: 138408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4227.972288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.546920] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4233.187333] LustreError: 138409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.307841] LustreError: 138408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4242.435880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.923054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4248.945852] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4258.846698] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4265.645811] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4273.438338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4273.640164] LustreError: 140230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.805996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.481427] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4288.092187] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4292.604168] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4292.801790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4296.658837] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4296.834696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4305.545154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4305.704390] Lustre: lustre-OST0001: new disk, initializing [ 4305.712038] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4305.719806] Lustre: 141778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4305.816132] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4310.711664] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4312.170841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4312.179208] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4312.250512] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4317.724522] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4317.933590] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4321.951309] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4322.194482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4326.832317] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95ff435a0800.ost_server_uuid 50 [ 4328.076362] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95ff435a0800.ost_server_uuid in IDLE state after 0 sec [ 4338.161399] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4338.162435] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.175385] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4338.175392] Lustre: Skipped 2 previous similar messages [ 4338.218042] Lustre: Skipped 2 previous similar messages [ 4342.301391] Lustre: server umount lustre-OST0001 complete [ 4343.296316] LustreError: 140232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.335892] LustreError: 140232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4348.392775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.399944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4348.403678] Lustre: Skipped 1 previous similar message [ 4348.418796] Lustre: Skipped 2 previous similar messages [ 4352.542243] Lustre: server umount lustre-OST0000 complete [ 4358.626987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.655519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.662924] Lustre: Skipped 1 previous similar message [ 4362.941837] Lustre: server umount lustre-MDT0000 complete [ 4363.755921] LustreError: 138405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.788217] LustreError: 138405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4366.579847] LustreError: 138391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552798 with bad export cookie 18297575969740952744 [ 4366.595729] LustreError: 138391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4366.599520] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4366.946734] Lustre: server umount lustre-MDT0001 complete [ 4386.592771] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4390.174666] Key type lgssc unregistered [ 4390.523852] LNet: 144786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4390.534408] LNetError: 144786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4390.566599] LNet: Removed LNI 192.168.206.152@tcp [ 4391.607311] Key type .llcrypt unregistered [ 4391.609577] Key type ._llcrypt unregistered [ 4429.367804] Key type ._llcrypt registered [ 4429.373763] Key type .llcrypt registered [ 4429.539692] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 4447.458817] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4448.896480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4449.204736] alg: No test for adler32 (adler32-zlib) [ 4450.523558] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 4451.020185] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4452.880244] Key type lgssc registered [ 4454.580473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4494.419361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4494.446283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.748475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4495.792926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4495.868359] Lustre: lustre-MDT0000: new disk, initializing [ 4495.957285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.969886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4501.158758] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4512.447074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.638075] Lustre: 148944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4512.717523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4512.724790] Lustre: Skipped 1 previous similar message [ 4512.822931] Lustre: lustre-MDT0001: new disk, initializing [ 4513.018839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4513.093786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4513.110991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4518.682889] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4530.272238] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4536.324848] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4544.590042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4544.929386] Lustre: lustre-OST0000: new disk, initializing [ 4544.938507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4544.954493] Lustre: 150914:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4545.112384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4551.905478] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4554.326145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4554.335163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4554.450336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4563.153146] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4567.661986] Lustre: Failing over lustre-OST0000 [ 4567.798538] Lustre: server umount lustre-OST0000 complete [ 4586.338107] Lustre: 147165:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553002/real 1787553002] req@0000000090640060 x1874385051880320/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787553018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4586.369294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4587.068925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.301164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4587.312718] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4588.320894] Lustre: 147168:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553002/real 1787553002] req@00000000e25b69fc x1874385051880576/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787553018 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4588.338760] Lustre: 147168:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4591.074804] Lustre: 147166:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553007/real 1787553007] req@0000000042f6503f x1874385051881856/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787553023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4592.540000] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4595.920880] Lustre: Failing over lustre-MDT0000 [ 4596.199783] Lustre: server umount lustre-MDT0000 complete [ 4596.692411] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4596.714701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.743966] Lustre: Skipped 3 previous similar messages [ 4597.797380] Lustre: 147166:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553012/real 1787553012] req@00000000a92999c8 x1874385051882880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787553028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4597.811863] Lustre: 147166:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4601.790427] LustreError: 148950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4601.822329] LustreError: 148950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4601.888344] Lustre: 147166:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553018/real 1787553018] req@000000004d741e8f x1874385051884928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787553034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4601.922633] Lustre: 147166:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4602.862061] LustreError: 148949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.885491] LustreError: 148949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4606.907524] LustreError: 148951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4612.002878] Lustre: 147166:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553028/real 1787553028] req@00000000b374f574 x1874385051891584/t0(0) o400->MGC192.168.206.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787553044 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4612.024779] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4612.041554] LustreError: 148950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4612.051100] LustreError: 148950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4615.730848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.164901] LustreError: 148951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.185947] LustreError: 148951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4622.306357] LustreError: 147164:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b50fef61 x1874385051895936/t0(0) o250->MGC192.168.206.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4622.528547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4622.599714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.636695] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4626.695719] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4627.393872] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4627.967498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4627.988839] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4628.023283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4636.153582] Lustre: server umount lustre-OST0000 complete [ 4643.298871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4643.302173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.322530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4643.329156] Lustre: Skipped 1 previous similar message [ 4643.346936] Lustre: Skipped 1 previous similar message [ 4648.418964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4648.432418] Lustre: Skipped 1 previous similar message [ 4650.567805] Lustre: server umount lustre-MDT0000 complete [ 4653.537320] LustreError: 148950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4653.557659] LustreError: 148950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4654.180309] LustreError: 148936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553086 with bad export cookie 7875724634010537701 [ 4654.190347] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4654.198921] LustreError: 148936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4664.014340] Lustre: server umount lustre-MDT0001 complete [ 4675.181188] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4678.667054] Key type lgssc unregistered [ 4679.038460] LNet: 154573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4679.056783] LNetError: 154573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4680.103444] LNet: Removed LNI 192.168.206.152@tcp [ 4681.290746] Key type .llcrypt unregistered [ 4681.294293] Key type ._llcrypt unregistered [ 4717.575377] Key type ._llcrypt registered [ 4717.577542] Key type .llcrypt registered [ 4717.669728] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4718.472376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4718.585881] alg: No test for adler32 (adler32-zlib) [ 4719.665868] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 4719.963916] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4721.632186] Key type lgssc registered [ 4722.623921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4729.620374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4729.642234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.168174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4735.134385] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4741.601748] LustreError: 156097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.628900] LustreError: 156097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4743.022883] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4746.724561] LustreError: 156096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4746.726912] LustreError: 156125:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4746.749509] LustreError: 156125:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4746.756564] LustreError: 156125:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4747.030402] Lustre: server umount lustre-MDT0000 complete [ 4761.063713] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 4770.366463] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4803.496405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.868877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4803.907357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4804.088744] Lustre: lustre-MDT0000: new disk, initializing [ 4804.188275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.206394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4808.062847] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4819.563427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4819.705874] Lustre: 159428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4819.796321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4819.800547] Lustre: Skipped 1 previous similar message [ 4819.877321] Lustre: lustre-MDT0001: new disk, initializing [ 4820.007406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4820.075352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4820.101407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4825.841539] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4834.808153] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4839.723691] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4845.641704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.979354] Lustre: lustre-OST0000: new disk, initializing [ 4845.984189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4845.992703] Lustre: 161397:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4846.067296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.082936] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4853.383453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4853.402748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4853.503573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4861.234909] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4865.870579] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4867.684911] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:34:59 (1787553299) [ 5094.368793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.369198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.386781] Lustre: Skipped 2 previous similar messages [ 5094.398554] Lustre: Skipped 2 previous similar messages [ 5099.490068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5099.497550] Lustre: Skipped 2 previous similar messages [ 5102.315680] Lustre: server umount lustre-MDT0000 complete [ 5104.610131] LustreError: 159434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.641629] LustreError: 159434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5106.141770] LustreError: 161396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553538 with bad export cookie 12613932613586421027 [ 5106.142838] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5106.155428] LustreError: 161396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5109.729308] LustreError: 163174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5109.730221] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.777423] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5111.779587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5112.500549] Lustre: server umount lustre-MDT0001 complete [ 5122.896826] Lustre: server umount lustre-OST0000 complete [ 5134.385705] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 5144.891848] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5178.323371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5178.501932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5178.525468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5178.581385] Lustre: lustre-MDT0000: new disk, initializing [ 5178.634419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.646956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5182.805517] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5193.238360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.340318] Lustre: 166052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5193.380369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5193.383732] Lustre: Skipped 1 previous similar message [ 5193.444736] Lustre: lustre-MDT0001: new disk, initializing [ 5193.554372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5193.592557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5193.613516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5197.977350] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5206.914114] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5212.828471] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5220.298949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.620769] Lustre: lustre-OST0000: new disk, initializing [ 5220.624699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5220.630146] Lustre: 168019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5220.726962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.211777] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5230.686432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5230.714093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5230.841202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5237.342432] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5241.629177] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5241.861440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5246.755846] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5247.142742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5251.556207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5251.559433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.564455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5251.574646] Lustre: Skipped 1 previous similar message [ 5251.593602] Lustre: Skipped 1 previous similar message [ 5261.806880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5261.818288] Lustre: Skipped 3 previous similar messages [ 5263.328201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5263.502648] Lustre: server umount lustre-OST0000 complete [ 5270.497585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5270.515034] LustreError: Skipped 1 previous similar message [ 5270.524944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.665300] Lustre: server umount lustre-MDT0000 complete [ 5276.843259] LustreError: 166044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553709 with bad export cookie 12613932613586579864 [ 5276.846288] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5276.860344] LustreError: 166044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5277.077176] Lustre: server umount lustre-MDT0001 complete [ 5285.488645] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:41:56 (1787553716) [ 5302.914609] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 5313.596148] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5348.159855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.418579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5348.467846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5348.544611] Lustre: lustre-MDT0000: new disk, initializing [ 5348.614199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.626168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5351.924971] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5360.376956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.455633] Lustre: 172780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5360.690894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5363.802834] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5372.418300] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5377.723352] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5383.773765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5384.034933] Lustre: lustre-OST0000: new disk, initializing [ 5384.037772] Lustre: Skipped 1 previous similar message [ 5384.040649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5384.044166] Lustre: Skipped 2 previous similar messages [ 5384.048809] Lustre: 174754:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5389.722683] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5393.986610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5393.991806] Lustre: Skipped 1 previous similar message [ 5394.000947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5394.055269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5397.557375] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5404.651510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5404.655728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.663355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5404.673990] Lustre: Skipped 2 previous similar messages [ 5404.687810] Lustre: Skipped 3 previous similar messages [ 5409.147556] Lustre: server umount lustre-OST0000 complete [ 5414.884455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.901200] Lustre: Skipped 1 previous similar message [ 5418.411683] Lustre: server umount lustre-MDT0000 complete [ 5420.007636] LustreError: 172784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5420.029475] LustreError: 172784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5421.444448] LustreError: 172772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787553853 with bad export cookie 12613932613586580746 [ 5421.446801] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5421.454094] LustreError: 172772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5421.754728] Lustre: server umount lustre-MDT0001 complete [ 5428.352354] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:44:20 (1787553860) [ 5441.208842] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 5451.073924] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5489.160186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.585813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5489.655866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5489.798469] Lustre: lustre-MDT0000: new disk, initializing [ 5489.939017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.948395] Lustre: Skipped 2 previous similar messages [ 5489.983485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5495.404487] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5506.082893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.181705] Lustre: 179075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5506.512254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5510.732232] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5519.687320] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5525.412937] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5531.181595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5531.501497] Lustre: 181053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5537.432691] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5538.020075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5545.972245] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5553.521259] Lustre: server umount lustre-OST0000 complete [ 5557.729964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5557.736075] LustreError: Skipped 1 previous similar message [ 5557.738465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5557.745437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5557.749414] Lustre: Skipped 2 previous similar messages [ 5565.986862] Lustre: server umount lustre-MDT0000 complete [ 5568.992162] Lustre: 155408:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787553985/real 1787553985] req@00000000d38c6923 x1874385334711168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787554001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5569.003875] LustreError: 180245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5569.045057] Lustre: 155408:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5569.072661] LustreError: 180245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5569.420553] LustreError: 179065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554001 with bad export cookie 12613932613586581831 [ 5569.422551] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5569.434988] LustreError: 179065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5569.735902] Lustre: server umount lustre-MDT0001 complete [ 5580.207318] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 5583.128163] Key type lgssc unregistered [ 5583.370679] LNet: 182911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5583.378694] LNetError: 182911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5583.387774] LNet: Removed LNI 192.168.206.152@tcp [ 5584.183596] Key type .llcrypt unregistered [ 5584.185110] Key type ._llcrypt unregistered [ 5611.079581] Key type ._llcrypt registered [ 5611.083060] Key type .llcrypt registered [ 5611.190389] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5612.466487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5612.513393] alg: No test for adler32 (adler32-zlib) [ 5613.504274] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 5613.743887] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 5615.448192] Key type lgssc registered [ 5616.606589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5624.566505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5624.594571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.132396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.313916] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5636.580734] LustreError: 184328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5636.598544] LustreError: 184328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5638.751057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.160582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5643.779709] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5651.553862] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5658.114943] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5665.325651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.580960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5670.795851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5673.086528] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5680.392448] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5701.603767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.616607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.632794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.698235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.703114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.711176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.729708] Lustre: Skipped 1 previous similar message [ 5707.479940] Lustre: server umount lustre-OST0000 complete [ 5715.941143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5715.943605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.950257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5715.957087] Lustre: Skipped 1 previous similar message [ 5715.969146] Lustre: Skipped 1 previous similar message [ 5717.289768] Lustre: server umount lustre-MDT0000 complete [ 5720.882841] LustreError: 184311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554153 with bad export cookie 6998458812626176777 [ 5720.885590] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5720.892883] LustreError: 184311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5721.263369] Lustre: server umount lustre-MDT0001 complete [ 5730.850726] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 5733.574690] Key type lgssc unregistered [ 5734.003625] LNet: 187889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5734.011541] LNetError: 187889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5734.039782] LNet: Removed LNI 192.168.206.152@tcp [ 5734.911055] Key type .llcrypt unregistered [ 5734.912730] Key type ._llcrypt unregistered [ 5758.272526] Key type ._llcrypt registered [ 5758.276215] Key type .llcrypt registered [ 5758.402916] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5759.313790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5759.364326] alg: No test for adler32 (adler32-zlib) [ 5760.427993] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 5760.659849] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 5762.304502] Key type lgssc registered [ 5763.432486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5770.481911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5770.502274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.964065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5775.949070] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5782.498593] LustreError: 189305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5782.513269] LustreError: 189305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5782.909652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.273101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5786.674700] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5793.006983] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5797.992218] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5803.686931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5803.928679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5809.951384] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5816.588675] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5817.341451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5823.458334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5823.469503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.488440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5824.473631] Lustre: lustre-OST0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 5827.556635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.559700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5827.579471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5827.581925] Lustre: Skipped 1 previous similar message [ 5829.036366] Lustre: server umount lustre-OST0000 complete [ 5836.082739] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff95ff433a5000.ost_server_uuid 50 [ 5837.836771] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5855.206361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5855.218865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.232435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5857.767430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.535939] Lustre: server umount lustre-MDT0000 complete [ 5861.477799] LustreError: 190040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554293 with bad export cookie 2011609208981234719 [ 5861.484625] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5861.491545] LustreError: 190040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5861.772565] Lustre: server umount lustre-MDT0001 complete [ 5868.029852] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:51:40 (1787554300) [ 5873.349575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5873.724380] LustreError: 193219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.806858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.551416] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5879.267846] LustreError: 193220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.392051] LustreError: 193219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.849836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5885.273261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5889.040904] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5895.546391] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5900.328132] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5905.877717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.163878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.214466] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5916.768704] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5920.496635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5921.962422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5922.124974] Lustre: lustre-OST0001: new disk, initializing [ 5922.127882] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5922.133077] Lustre: 196061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5922.230255] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5926.896325] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5931.561759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5931.567579] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5931.627140] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5933.990271] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5936.098214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.102988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.120650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.130734] Lustre: Skipped 3 previous similar messages [ 5942.115656] Lustre: server umount lustre-OST0000 complete [ 5944.768276] LustreError: 195041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.875751] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5947.045457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5950.533783] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5950.745630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5955.008789] LustreError: 196660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.026113] LustreError: 196660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5967.332985] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5967.333709] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.343183] LustreError: Skipped 2 previous similar messages [ 5967.344434] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5967.351501] Lustre: Skipped 2 previous similar messages [ 5967.371269] Lustre: Skipped 4 previous similar messages [ 5969.999942] Lustre: server umount lustre-OST0001 complete [ 5977.571522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5979.449689] Lustre: server umount lustre-MDT0000 complete [ 5982.501280] LustreError: 193201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554414 with bad export cookie 2011609208981235790 [ 5982.510538] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5982.516835] LustreError: 193201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5982.746674] Lustre: server umount lustre-MDT0001 complete [ 6004.210448] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:53:55 (1787554435) [ 6010.159856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.406085] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6010.447530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6010.673331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.502445] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6022.308226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.460607] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6022.514098] Lustre: 199500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6027.658653] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6034.492892] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6039.916472] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6046.428317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6046.636863] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6052.088662] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6055.062555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6058.288723] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6063.888064] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6063.936285] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6069.151179] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6070.398640] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6075.499959] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6080.482964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6080.489212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.499281] Lustre: Skipped 2 previous similar messages [ 6080.505364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6080.512576] Lustre: Skipped 2 previous similar messages [ 6092.256447] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6092.429319] Lustre: server umount lustre-OST0000 complete [ 6096.357211] LustreError: 201349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.371184] LustreError: 201349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6113.403622] Lustre: server umount lustre-OST0001 complete [ 6120.417436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6120.428138] LustreError: Skipped 3 previous similar messages [ 6120.433535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6120.457761] Lustre: Skipped 3 previous similar messages [ 6120.461658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6120.464077] Lustre: Skipped 10 previous similar messages [ 6132.192129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6132.410841] Lustre: server umount lustre-MDT0000 complete [ 6135.778551] LustreError: 201379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554568 with bad export cookie 2011609208981237197 [ 6135.784327] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6135.790800] LustreError: 201379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6136.113302] Lustre: server umount lustre-MDT0001 complete [ 6155.708343] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:56:27 (1787554587) [ 6170.591856] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 6179.919723] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 6210.775063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.026215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6211.065499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6211.146279] Lustre: lustre-MDT0000: new disk, initializing [ 6211.244769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.255875] Lustre: Skipped 3 previous similar messages [ 6211.284630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6215.408557] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6225.267795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.346440] Lustre: 207594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6225.405442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6225.410968] Lustre: Skipped 1 previous similar message [ 6225.503917] Lustre: lustre-MDT0001: new disk, initializing [ 6225.610189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6225.620945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6229.640702] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6239.139444] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6245.992328] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6253.476111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6253.744469] Lustre: lustre-OST0000: new disk, initializing [ 6253.750211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6253.760752] Lustre: 209564:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6259.683219] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6263.891448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6263.903890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6264.042812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6269.520248] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6274.663722] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6274.943966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6280.149782] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6280.478795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6284.794667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.803344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6284.825884] Lustre: Skipped 2 previous similar messages [ 6284.829490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6284.859549] Lustre: Skipped 7 previous similar messages [ 6288.924461] Lustre: server umount lustre-OST0000 complete [ 6298.928692] Lustre: server umount lustre-MDT0000 complete [ 6300.137243] LustreError: 208596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.155857] LustreError: 208596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6302.807936] LustreError: 207587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554735 with bad export cookie 2011609208981238989 [ 6302.813555] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6302.817622] LustreError: 207587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6303.278986] Lustre: server umount lustre-MDT0001 complete [ 6311.626276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.573197] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6325.511127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.539726] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6338.491263] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6344.344516] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6350.943875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6351.318713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.323288] Lustre: Skipped 4 previous similar messages [ 6357.102470] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6366.200916] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6372.526109] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6372.846983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6377.922766] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6378.175622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6386.547343] Lustre: server umount lustre-OST0000 complete [ 6392.806714] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6396.984800] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6425.291518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6431.797793] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6436.915133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.925901] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6436.927056] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.083438] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6443.867947] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6452.195040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6452.195211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.208238] LustreError: Skipped 2 previous similar messages [ 6452.208981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6452.222916] Lustre: Skipped 5 previous similar messages [ 6452.243199] Lustre: Skipped 7 previous similar messages [ 6453.881327] Lustre: server umount lustre-OST0000 complete [ 6464.678422] Lustre: server umount lustre-MDT0000 complete [ 6467.553428] LustreError: 211781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6467.593601] LustreError: 211781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6469.269371] LustreError: 211761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554901 with bad export cookie 2011609208981239871 [ 6469.276736] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6469.279917] LustreError: 211761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6469.852883] Lustre: server umount lustre-MDT0001 complete [ 6480.295980] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:01:51 (1787554911) [ 6488.200956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6494.005358] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6503.598628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.161618] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6516.674562] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6522.239476] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6529.664063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6537.129785] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6545.584487] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6550.566996] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6550.777725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6555.233126] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6555.444722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6562.123202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.418922] Lustre: lustre-OST0001: new disk, initializing [ 6562.426531] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6562.441379] Lustre: 221131:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6569.487518] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6570.690688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6570.697627] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6570.878198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6579.711992] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6584.966221] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6585.174955] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6589.296608] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6589.512425] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6598.015303] Lustre: server umount lustre-OST0001 complete [ 6601.699509] LustreError: 219582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.729295] LustreError: 219582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6603.761191] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6603.966409] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6608.057180] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6608.269274] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6636.582904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.760852] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6636.765359] Lustre: Skipped 5 previous similar messages [ 6642.162445] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6642.185087] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6642.185339] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6642.212268] Lustre: Skipped 2 previous similar messages [ 6643.470224] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6652.174465] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6653.661760] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6670.304114] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6670.474103] Lustre: server umount lustre-OST0001 complete [ 6702.048253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6702.343750] Lustre: server umount lustre-OST0000 complete [ 6708.706060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6708.706704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.715266] LustreError: Skipped 7 previous similar messages [ 6708.723831] Lustre: Skipped 8 previous similar messages [ 6708.731252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6708.742377] Lustre: Skipped 18 previous similar messages [ 6712.897940] Lustre: server umount lustre-MDT0000 complete [ 6716.827684] LustreError: 217738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555149 with bad export cookie 2011609208981240837 [ 6716.835887] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6716.837900] LustreError: 217738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6731.232139] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6731.637636] Lustre: server umount lustre-MDT0001 complete [ 6758.128264] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:06:29 (1787555189) [ 6765.394837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.950424] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6766.009348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6770.656702] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6778.827909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.944167] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6778.970213] Lustre: 226378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6783.405945] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6791.291725] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6796.654628] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6804.364813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.588831] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6812.158287] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6819.926795] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6830.363924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.513957] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6837.435934] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6845.036349] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6846.642773] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6853.232867] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6853.563921] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6858.916576] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6859.235921] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6866.643570] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95ff4736e800.ost_server_uuid 50 [ 6868.592917] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95ff4736e800.ost_server_uuid in IDLE state after 0 sec [ 6869.599423] Lustre: Permanently deactivating lustre-OST0001 [ 6869.627516] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6873.922878] Lustre: Permanently reactivating lustre-OST0001 [ 6873.952223] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6873.961659] Lustre: Skipped 2 previous similar messages [ 6884.733315] Lustre: server umount lustre-OST0001 complete [ 6885.355185] LustreError: 228227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.380222] LustreError: 228227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 33 previous similar messages [ 6894.815799] Lustre: server umount lustre-OST0000 complete [ 6905.041192] Lustre: server umount lustre-MDT0000 complete [ 6908.874639] LustreError: 226372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555341 with bad export cookie 2011609208981242125 [ 6908.880623] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.888312] LustreError: 226372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6909.302720] Lustre: server umount lustre-MDT0001 complete [ 6932.122448] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:09:23 (1787555363) [ 6939.428758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6939.669614] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6939.688325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6939.694848] Lustre: Skipped 2 previous similar messages [ 6944.649181] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6953.898117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.988659] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6954.034911] Lustre: 233074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6959.130819] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6967.466312] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6972.976632] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6980.386080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6980.586323] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6980.617092] Lustre: Permanently deactivating lustre-OST0000 [ 6980.638178] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6987.785835] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6989.924866] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6995.389997] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7002.848543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7002.979090] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7010.498105] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7011.608654] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7019.742499] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7021.924947] Lustre: Permanently reactivating lustre-OST0000 [ 7021.941062] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7021.949681] Lustre: Skipped 2 previous similar messages [ 7029.897098] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7029.897772] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 000000009c6e0ce4, cur 1787555462 deadline 1787555452 last 1787555422 [ 7029.906886] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7029.945008] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7029.976662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7043.189806] Lustre: server umount lustre-OST0001 complete [ 7061.984189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7062.451254] Lustre: server umount lustre-OST0000 complete [ 7072.399260] Lustre: server umount lustre-MDT0000 complete [ 7075.930824] LustreError: 233066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555508 with bad export cookie 2011609208981243784 [ 7075.953730] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7076.514391] Lustre: server umount lustre-MDT0001 complete [ 7089.214945] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 7092.088743] Key type lgssc unregistered [ 7092.505671] LNet: 237961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7092.517291] LNetError: 237961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7092.535827] LNet: Removed LNI 192.168.206.152@tcp [ 7093.496193] Key type .llcrypt unregistered [ 7093.499078] Key type ._llcrypt unregistered [ 7123.101334] Key type ._llcrypt registered [ 7123.105352] Key type .llcrypt registered [ 7123.289968] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7124.643640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7124.715337] alg: No test for adler32 (adler32-zlib) [ 7126.206117] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 7126.649196] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 7128.504228] Key type lgssc registered [ 7129.756460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7140.447951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7140.473855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7141.915362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7146.276355] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7152.102572] LustreError: 239428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7152.139583] LustreError: 239428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7154.894713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.028132] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7155.049360] Lustre: 239415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7155.057869] Lustre: Permanently deactivating lustre-MDT0001 [ 7155.070309] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7155.116604] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7155.123367] LustreError: 239459:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7155.136988] LustreError: 239459:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7155.156353] LustreError: 239459:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 7155.345921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7160.148760] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7168.401258] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7174.162971] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7180.950758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7181.432683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7187.565017] LustreError: 241270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7187.585217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7188.651412] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7193.064305] LustreError: 241270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.109353] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7198.183648] LustreError: 241271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.928487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7204.235128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7209.477013] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7209.491801] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7209.905710] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7217.752831] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7221.744415] Lustre: Permanently reactivating lustre-MDT0001 [ 7221.762742] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7221.767436] Lustre: Skipped 2 previous similar messages [ 7226.866530] Lustre: lustre-MDT0001: Client d0556f81-ad54-4b27-8a98-594a0bbfc123 (at 192.168.206.52@tcp) reconnecting [ 7233.297334] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7234.662804] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7240.809383] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7241.150119] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7245.842695] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7246.027724] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7248.597456] Lustre: Permanently deactivating lustre-MDT0001 [ 7248.628448] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7248.644901] Lustre: Skipped 2 previous similar messages [ 7254.091732] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7265.763912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7265.767731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7265.785377] Lustre: Skipped 2 previous similar messages [ 7265.794817] Lustre: Skipped 1 previous similar message [ 7270.890725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7270.898834] Lustre: Skipped 5 previous similar messages [ 7271.406335] Lustre: server umount lustre-MDT0000 complete [ 7275.028548] LustreError: 239408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555707 with bad export cookie 13653974685339993320 [ 7275.030962] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7275.046930] LustreError: 239408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7275.369665] Lustre: server umount lustre-MDT0001 complete [ 7281.424463] Lustre: server umount lustre-OST0000 complete [ 7291.947887] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:15:23 (1787555723) [ 7302.649322] Lustre: server umount lustre-OST0001 complete [ 7308.541319] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 7318.130418] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7356.622993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.040195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7357.057758] Lustre: Skipped 2 previous similar messages [ 7357.115779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7357.273255] Lustre: lustre-MDT0000: new disk, initializing [ 7357.492420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.533263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7362.583849] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7373.296223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.406240] Lustre: 247698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7373.474846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7373.483274] Lustre: Skipped 1 previous similar message [ 7373.585584] Lustre: lustre-MDT0001: new disk, initializing [ 7373.700195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7373.758727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7373.770656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7377.767552] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7386.499594] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7392.222827] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7398.143959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7398.483747] Lustre: lustre-OST0000: new disk, initializing [ 7398.489601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7398.496402] Lustre: 249667:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7398.615847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7405.055916] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7405.104512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7405.118777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7405.219745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7413.663180] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7415.606647] LustreError: 247705:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7423.703818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7423.905776] Lustre: lustre-OST0001: new disk, initializing [ 7423.909546] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7423.915697] Lustre: 250781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7425.662247] LustreError: 247705:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7425.703929] LustreError: 250436:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7429.712581] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7430.225731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7430.235435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7430.398548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7435.720371] LustreError: 250436:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7439.429701] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7440.958071] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7445.995845] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7445.999735] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7446.001754] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7446.015199] Lustre: Skipped 2 previous similar messages [ 7446.021286] LustreError: Skipped 1 previous similar message [ 7451.115891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7451.138223] Lustre: Skipped 2 previous similar messages [ 7456.227764] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7456.238126] Lustre: Skipped 1 previous similar message [ 7457.248149] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7457.351769] Lustre: server umount lustre-OST0001 complete [ 7461.349062] LustreError: 250033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7461.373662] LustreError: 250033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7471.592394] LustreError: 249650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.616431] LustreError: 249650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7476.713858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7476.720640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7476.721636] Lustre: Skipped 1 previous similar message [ 7479.705269] Lustre: server umount lustre-OST0000 complete [ 7486.433749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7486.445714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.796244] Lustre: server umount lustre-MDT0000 complete [ 7492.067228] LustreError: 247710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.082932] LustreError: 247710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7493.445198] LustreError: 247689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555925 with bad export cookie 13653974685339995924 [ 7493.445761] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7493.453168] LustreError: 247689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7499.920667] Lustre: server umount lustre-MDT0001 complete [ 7511.175251] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 7513.900443] Key type lgssc unregistered [ 7514.190681] LNet: 252818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7514.210846] LNetError: 252818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7514.247855] LNet: Removed LNI 192.168.206.152@tcp [ 7515.197661] Key type .llcrypt unregistered [ 7515.200846] Key type ._llcrypt unregistered [ 7556.876823] Key type ._llcrypt registered [ 7556.880717] Key type .llcrypt registered [ 7557.003444] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7558.454391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7558.519375] alg: No test for adler32 (adler32-zlib) [ 7559.723503] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 7559.990606] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 7561.696724] Key type lgssc registered [ 7563.195947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7570.911483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7570.933330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7572.222704] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7572.257527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7572.422890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7576.551475] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7584.607599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.711858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7584.741850] Lustre: 255026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7585.022128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7589.211404] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7596.100797] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7600.877402] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7606.824155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7606.999167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7607.203420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.395559] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7617.146895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7621.091950] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7678.457819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7678.473445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7678.506413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7680.824753] Lustre: server umount lustre-OST0000 complete [ 7685.404972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7698.279847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7698.685437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7699.691441] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7699.711578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7699.716378] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7705.501959] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7712.772115] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7751.144975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7751.149907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.159370] LustreError: Skipped 2 previous similar messages [ 7751.177827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7751.178615] Lustre: Skipped 2 previous similar messages [ 7751.190054] Lustre: Skipped 2 previous similar messages [ 7756.260719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7756.265969] Lustre: Skipped 1 previous similar message [ 7756.467483] Lustre: server umount lustre-OST0000 complete [ 7759.331957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7759.337991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7759.349656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.382958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.361450] Lustre: server umount lustre-MDT0000 complete [ 7768.219994] LustreError: 255018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556200 with bad export cookie 17753843651556697150 [ 7768.227148] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7768.236066] LustreError: 255018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7768.643244] Lustre: server umount lustre-MDT0001 complete [ 7778.904580] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 7781.481247] Key type lgssc unregistered [ 7781.713125] LNet: 260452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7781.718008] LNetError: 260452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7782.761339] LNet: Removed LNI 192.168.206.152@tcp [ 7783.634754] Key type .llcrypt unregistered [ 7783.639339] Key type ._llcrypt unregistered [ 7806.487640] Key type ._llcrypt registered [ 7806.492123] Key type .llcrypt registered [ 7806.621970] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7807.239483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7807.277554] alg: No test for adler32 (adler32-zlib) [ 7808.498561] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 7808.847657] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 7810.553654] Key type lgssc registered [ 7811.250719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7821.480984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7821.523505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7823.125666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.274672] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7833.593768] LustreError: 261872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.619704] LustreError: 261872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7838.703581] LustreError: 261873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7838.955896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7839.451947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7844.548916] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7851.835367] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7858.189631] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7864.875319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7865.100402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7868.212348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7871.138988] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7877.576965] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7904.226354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7904.243834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.263506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7905.249399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7905.265063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7905.275867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7905.285530] Lustre: Skipped 1 previous similar message [ 7909.996204] Lustre: server umount lustre-OST0000 complete [ 7915.489722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7915.497641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7915.518293] Lustre: Skipped 1 previous similar message [ 7915.543308] Lustre: Skipped 1 previous similar message [ 7920.113540] Lustre: server umount lustre-MDT0000 complete [ 7920.612965] LustreError: 262600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.642508] LustreError: 262600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7924.237073] LustreError: 261852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556356 with bad export cookie 17606312641477415566 [ 7924.242871] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7924.254914] LustreError: 261852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7924.714703] Lustre: server umount lustre-MDT0001 complete [ 7935.598772] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 7938.055682] Key type lgssc unregistered [ 7938.343752] LNet: 266179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7938.349231] LNetError: 266179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7938.370821] LNet: Removed LNI 192.168.206.152@tcp [ 7939.234264] Key type .llcrypt unregistered [ 7939.238465] Key type ._llcrypt unregistered [ 7955.114531] Key type ._llcrypt registered [ 7955.116097] Key type .llcrypt registered [ 7955.195611] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7956.099830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7956.162892] alg: No test for adler32 (adler32-zlib) [ 7957.376706] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 7957.654619] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 7959.384176] Key type lgssc registered [ 7960.572719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7968.005533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7968.021174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7969.526574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7974.534517] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7978.977874] LustreError: 267401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7979.001528] LustreError: 267401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7982.503752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.936231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7987.120300] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7995.250691] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8001.071411] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8007.451420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.735559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.218238] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8013.864374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8020.595217] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8034.276242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8034.281078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8034.284027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8034.299659] Lustre: Skipped 1 previous similar message [ 8037.015372] Lustre: server umount lustre-OST0000 complete [ 8044.516813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.518182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8044.554025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8044.554032] Lustre: Skipped 1 previous similar message [ 8049.633865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8049.650882] Lustre: Skipped 2 previous similar messages [ 8054.756159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8054.770481] Lustre: Skipped 1 previous similar message [ 8055.266981] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8055.605756] Lustre: server umount lustre-MDT0000 complete [ 8059.878806] LustreError: 267400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8059.896693] LustreError: 267400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8060.230752] LustreError: 267380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556492 with bad export cookie 13709107804346652572 [ 8060.239035] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.252531] LustreError: 267380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8061.112222] Lustre: server umount lustre-MDT0001 complete [ 8074.127461] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 8077.805805] Key type lgssc unregistered [ 8078.213166] LNet: 271081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8078.224559] LNetError: 271081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8078.247142] LNet: Removed LNI 192.168.206.152@tcp [ 8079.089904] Key type .llcrypt unregistered [ 8079.093194] Key type ._llcrypt unregistered [ 8104.632455] Key type ._llcrypt registered [ 8104.634456] Key type .llcrypt registered [ 8104.708499] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 8105.764231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8105.786931] alg: No test for adler32 (adler32-zlib) [ 8106.762226] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 8106.975470] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 8108.832562] Key type lgssc registered [ 8110.579294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8119.306783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8119.350040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.768683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8125.997743] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8131.043427] LustreError: 272500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8131.058471] LustreError: 272500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8135.568966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.862870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8140.560151] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8147.947985] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8153.849816] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8162.391455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8162.749571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8165.889867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8168.695877] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8177.058568] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8207.849712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8207.852473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8207.852620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8207.891568] Lustre: Skipped 1 previous similar message [ 8207.932142] Lustre: Skipped 1 previous similar message [ 8212.967505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8212.977535] Lustre: Skipped 1 previous similar message [ 8213.208603] Lustre: server umount lustre-OST0000 complete [ 8218.082760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8218.084700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8218.100450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8218.111178] Lustre: Skipped 1 previous similar message [ 8218.121191] Lustre: Skipped 1 previous similar message [ 8223.203992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8223.217352] Lustre: Skipped 1 previous similar message [ 8223.524657] Lustre: server umount lustre-MDT0000 complete [ 8227.590138] LustreError: 276056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556659 with bad export cookie 8997666320984071407 [ 8227.592971] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8227.603263] LustreError: 276056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8228.002664] Lustre: server umount lustre-MDT0001 complete [ 8240.901641] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 8243.903374] Key type lgssc unregistered [ 8244.188809] LNet: 276811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8244.205684] LNetError: 276811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8244.229126] LNet: Removed LNI 192.168.206.152@tcp [ 8245.052924] Key type .llcrypt unregistered [ 8245.055973] Key type ._llcrypt unregistered [ 8263.329297] Key type ._llcrypt registered [ 8263.332169] Key type .llcrypt registered [ 8263.439560] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 8264.490251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8264.504473] alg: No test for adler32 (adler32-zlib) [ 8265.736190] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 8265.937194] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 8267.680193] Key type lgssc registered [ 8269.020934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8276.745830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8276.763831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.386693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8282.765404] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8288.743816] LustreError: 278048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8288.767058] LustreError: 278048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8293.366290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.845764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8299.101894] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8306.467632] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8314.316250] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8324.403535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8324.658855] LustreError: 279896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8325.047318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8330.246235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8333.654961] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8342.540680] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8354.787743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8354.790114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.793149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8354.812317] Lustre: Skipped 1 previous similar message [ 8359.909552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8359.928633] Lustre: Skipped 1 previous similar message [ 8360.867699] Lustre: server umount lustre-OST0000 complete [ 8365.537410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8365.550893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.561830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.565747] Lustre: Skipped 1 previous similar message [ 8370.148447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.149374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.171288] Lustre: Skipped 1 previous similar message [ 8371.632755] Lustre: server umount lustre-MDT0000 complete [ 8375.274831] LustreError: 278058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.330667] LustreError: 278058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8376.203909] LustreError: 278014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556808 with bad export cookie 99610345652034317 [ 8376.216184] LustreError: 278014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8376.219361] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8377.258327] Lustre: server umount lustre-MDT0001 complete [ 8391.731198] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 8395.196867] Key type lgssc unregistered [ 8395.625242] LNet: 281756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8395.645155] LNetError: 281756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8395.667880] LNet: Removed LNI 192.168.206.152@tcp [ 8396.818696] Key type .llcrypt unregistered [ 8396.824048] Key type ._llcrypt unregistered [ 8406.161167] Key type ._llcrypt registered [ 8406.162742] Key type .llcrypt registered [ 8406.240604] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8422.035791] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 8435.367257] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 8436.081796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8436.155094] alg: No test for adler32 (adler32-zlib) [ 8437.201132] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 8437.423167] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 8439.120165] Key type lgssc registered [ 8440.315352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8475.341158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8475.387220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8476.633746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8476.661768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8476.728830] Lustre: lustre-MDT0000: new disk, initializing [ 8476.844749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8476.862121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8482.147910] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8494.670309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.854109] Lustre: 285301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8494.964766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8494.974682] Lustre: Skipped 1 previous similar message [ 8495.129237] Lustre: lustre-MDT0001: new disk, initializing [ 8495.290560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8495.357291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8495.388846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8500.902753] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8511.896962] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8518.045575] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8524.648554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8524.850990] Lustre: lustre-OST0000: new disk, initializing [ 8524.854546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8524.859554] Lustre: 287271:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8524.944578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8530.985916] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8534.577623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8534.584065] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8534.738025] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8539.724487] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8544.080554] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8544.265210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8549.183382] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8549.439815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8555.499562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8555.508120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8555.524965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8555.526949] LustreError: Skipped 1 previous similar message [ 8557.700452] Lustre: server umount lustre-OST0000 complete [ 8562.145409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8562.152299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.156891] Lustre: Skipped 1 previous similar message [ 8562.160797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8562.162941] Lustre: Skipped 1 previous similar message [ 8565.731637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8565.734764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8568.024234] Lustre: server umount lustre-MDT0000 complete [ 8573.113418] LustreError: 285295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557005 with bad export cookie 17802053266799732163 [ 8573.117614] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8573.140033] LustreError: 285295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8573.938400] Lustre: server umount lustre-MDT0001 complete [ 8583.493234] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:36:55 (1787557015) [ 8589.726286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.172447] LustreError: 289682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8590.192473] LustreError: 289682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8590.236773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.142920] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8595.432931] LustreError: 289683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.547829] LustreError: 289682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8602.660082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.134652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8607.162745] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8614.809019] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8620.930284] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8628.329425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8628.715700] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8628.730920] Lustre: Skipped 1 previous similar message [ 8628.918270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8637.238411] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8644.068333] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8654.817442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8654.825261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8654.834244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8659.249850] Lustre: server umount lustre-OST0000 complete [ 8664.552082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8664.559888] LustreError: Skipped 1 previous similar message [ 8664.562496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8664.569814] Lustre: Skipped 1 previous similar message [ 8664.580989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8664.593444] Lustre: Skipped 2 previous similar messages [ 8670.436629] Lustre: server umount lustre-MDT0000 complete [ 8671.203803] LustreError: 289679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8671.238484] LustreError: 289679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8676.315382] LustreError: 289663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557108 with bad export cookie 17802053266799733052 [ 8676.324568] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8676.556305] Lustre: server umount lustre-MDT0001 complete [ 8690.424570] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 8695.147751] Key type lgssc unregistered [ 8695.534214] LNet: 293223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8695.562811] LNetError: 293223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8695.590911] LNet: Removed LNI 192.168.206.152@tcp [ 8696.678788] Key type .llcrypt unregistered [ 8696.683259] Key type ._llcrypt unregistered [ 8725.173386] Key type ._llcrypt registered [ 8725.179445] Key type .llcrypt registered [ 8725.358756] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 8742.131153] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 8743.296071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8743.441454] alg: No test for adler32 (adler32-zlib) [ 8744.718572] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 8745.008439] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 8746.720155] Key type lgssc registered [ 8748.000744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8797.674276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8797.723574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.044863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8799.069221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8799.182551] Lustre: lustre-MDT0000: new disk, initializing [ 8799.261427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8799.276051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8804.780070] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8817.542430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.632098] Lustre: 297053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8817.675280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8817.680403] Lustre: Skipped 1 previous similar message [ 8817.778483] Lustre: lustre-MDT0001: new disk, initializing [ 8817.902342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8817.965610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8817.984979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8823.055450] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8832.665844] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8838.790730] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8845.295238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8845.599541] Lustre: lustre-OST2710: new disk, initializing [ 8845.603534] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8845.611108] Lustre: 299020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8845.733452] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8851.022409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8851.038295] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8851.201652] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8853.134560] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8864.541110] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8872.722697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8873.043736] Lustre: lustre-OST03e8: new disk, initializing [ 8873.053667] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8873.068538] Lustre: 300082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8873.254607] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8880.158609] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8880.274860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8880.298633] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8880.462414] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8888.448649] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8895.030513] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8895.243205] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8900.091607] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8900.269301] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8905.284277] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8905.524643] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8910.271163] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8910.582511] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8915.429585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8915.447757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.480787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8916.464189] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.470408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8916.483983] Lustre: Skipped 2 previous similar messages [ 8916.498069] Lustre: Skipped 3 previous similar messages [ 8917.508528] Lustre: server umount lustre-MDT0000 complete [ 8921.547055] LustreError: 297044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557353 with bad export cookie 8745051522637735799 [ 8921.552337] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8921.562993] LustreError: 297044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8921.593923] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.601914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8921.611893] Lustre: Skipped 1 previous similar message [ 8921.641434] Lustre: Skipped 1 previous similar message [ 8922.152920] Lustre: server umount lustre-MDT0001 complete [ 8928.864773] Lustre: server umount lustre-OST2710 complete [ 8933.089956] Lustre: server umount lustre-OST03e8 complete [ 8951.432840] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 8960.392054] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 8998.454293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.869549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8998.923388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8999.145871] Lustre: lustre-MDT0000: new disk, initializing [ 8999.309491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8999.395913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9004.984636] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9016.003379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.147068] Lustre: 305329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9016.201519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9016.205717] Lustre: Skipped 1 previous similar message [ 9016.273896] Lustre: lustre-MDT0001: new disk, initializing [ 9016.362062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9016.397123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9016.418218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9021.469362] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9032.150457] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9038.834748] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9046.280871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9046.558426] Lustre: lustre-OST0000: new disk, initializing [ 9046.567071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9046.580978] Lustre: 307299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9046.790979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9052.729597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9052.747950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9052.898073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9053.319549] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9064.140359] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9068.646169] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9068.927626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9073.673594] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9073.967519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9078.753367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9078.754513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.770193] LustreError: Skipped 1 previous similar message [ 9078.783512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9078.786743] Lustre: Skipped 1 previous similar message [ 9078.816315] Lustre: Skipped 1 previous similar message [ 9083.873708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9083.877699] Lustre: Skipped 1 previous similar message [ 9090.016169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9090.459660] Lustre: server umount lustre-OST0000 complete [ 9098.219974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9098.228554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9098.243516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9098.256765] Lustre: Skipped 2 previous similar messages [ 9100.734858] Lustre: server umount lustre-MDT0000 complete [ 9104.357427] LustreError: 305341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9104.392054] LustreError: 305341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9105.178903] LustreError: 305321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557537 with bad export cookie 8745051522637737486 [ 9105.198443] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9105.909819] Lustre: server umount lustre-MDT0001 complete [ 9115.447419] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:45:46 (1787557546) [ 9117.476235] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9119.481544] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:45:50 (1787557550) [ 9123.390405] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 9150.466541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9150.852765] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9150.869738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9151.003609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9155.145477] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9165.815468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.910854] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9165.934784] Lustre: 311162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9170.846814] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9195.438346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9195.641323] Lustre: Denying initial registration attempt from nid 192.168.206.152@tcp, specified as failover [ 9195.649690] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9195.676909] LustreError: 312617:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9195.693107] LustreError: 312617:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9195.697889] LustreError: 312617:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9195.769123] Lustre: server umount lustre-OST0000 complete [ 9195.771602] LustreError: 312617:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9203.174332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9203.192735] Lustre: Skipped 1 previous similar message [ 9203.201254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9203.206785] Lustre: Skipped 3 previous similar messages [ 9208.957082] Lustre: server umount lustre-MDT0000 complete [ 9213.420896] LustreError: 311955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9213.462977] LustreError: 311955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9214.594680] LustreError: 311154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557646 with bad export cookie 8745051522637738368 [ 9214.607906] LustreError: 311154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9214.614198] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9215.059771] Lustre: server umount lustre-MDT0001 complete [ 9231.944409] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 9235.343942] Key type lgssc unregistered [ 9235.580592] LNet: 313742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9235.594220] LNetError: 313742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9236.647793] LNet: Removed LNI 192.168.206.152@tcp [ 9237.400067] Key type .llcrypt unregistered [ 9237.401956] Key type ._llcrypt unregistered [ 9249.620593] Key type ._llcrypt registered [ 9249.623751] Key type .llcrypt registered [ 9249.711293] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 9250.796693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9250.819403] alg: No test for adler32 (adler32-zlib) [ 9251.904896] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 9252.103443] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 9253.802377] Key type lgssc registered [ 9255.311281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9308.203626] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 9316.961717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9317.009774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.485119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9318.546294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9318.844406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9324.257456] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9334.918837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.129480] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9335.225788] Lustre: 316514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9335.670978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9340.417175] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9347.174937] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9353.215925] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9373.483030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9373.765941] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9374.015320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9379.970489] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9387.263476] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9392.621409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9392.643102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9392.664392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9393.124594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9395.991843] Lustre: server umount lustre-OST0000 complete [ 9402.340710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9402.345322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9402.359100] Lustre: Skipped 1 previous similar message [ 9402.364698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9402.376123] Lustre: Skipped 2 previous similar messages [ 9403.362948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9403.370358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9403.375332] Lustre: Skipped 1 previous similar message [ 9405.754215] Lustre: server umount lustre-MDT0000 complete [ 9409.356306] LustreError: 316506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557841 with bad export cookie 17077535853063298300 [ 9409.360198] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9409.370394] LustreError: 316506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9409.883213] Lustre: server umount lustre-MDT0001 complete [ 9421.489713] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 9424.375682] Key type lgssc unregistered [ 9424.607198] LNet: 320091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9424.636224] LNetError: 320091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9424.661873] LNet: Removed LNI 192.168.206.152@tcp [ 9425.591667] Key type .llcrypt unregistered [ 9425.596306] Key type ._llcrypt unregistered [ 9451.681300] Key type ._llcrypt registered [ 9451.682967] Key type .llcrypt registered [ 9451.884049] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 9453.383395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9453.532105] alg: No test for adler32 (adler32-zlib) [ 9454.663847] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 9455.063553] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 9456.824268] Key type lgssc registered [ 9458.410978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9470.392705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9470.416736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.885954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9476.196409] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9482.211078] LustreError: 321512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9482.241070] LustreError: 321512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9487.103652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9487.337061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9487.484802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9491.972319] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9499.626850] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9505.569557] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9514.353481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9514.789611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9520.845088] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9527.388503] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9540.763958] Lustre: server umount lustre-MDT0000 complete [ 9542.637413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9542.647067] LustreError: 321531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9542.658200] Lustre: Skipped 1 previous similar message [ 9542.683425] LustreError: 321531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9544.266751] LustreError: 321492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557976 with bad export cookie 11751545991671608058 [ 9544.281996] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9544.288333] LustreError: 321492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9545.201915] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.52@tcp (stopping) [ 9545.201937] LustreError: 321508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9547.744987] LustreError: 322724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9547.746089] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9547.771715] Lustre: Skipped 1 previous similar message [ 9547.780750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9549.795206] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9550.753600] Lustre: server umount lustre-MDT0001 complete [ 9557.244913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9565.581588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.251125] LustreError: 320821:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000055d2fc12 x1874390298951040/t0(0) o250->MGC192.168.206.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9569.906552] LustreError: 323345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9569.924127] LustreError: 323345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9570.062395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9570.156542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9575.207223] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9579.243656] LustreError: 325122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9579.246180] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9579.263907] LustreError: 325122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9583.394367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.767629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9587.733938] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9589.244720] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9595.104302] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9596.509830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9601.794935] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9603.125606] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9609.703390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9609.705902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9609.707390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9609.707396] Lustre: Skipped 1 previous similar message [ 9609.740663] Lustre: Skipped 1 previous similar message [ 9613.470693] Lustre: server umount lustre-OST0000 complete [ 9619.944197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.945876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9619.945974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9619.953307] Lustre: Skipped 1 previous similar message [ 9619.977333] Lustre: Skipped 2 previous similar messages [ 9623.278588] Lustre: server umount lustre-MDT0000 complete [ 9625.060872] LustreError: 325124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9625.087403] LustreError: 325124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9626.493668] LustreError: 321492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558058 with bad export cookie 11751545991671615254 [ 9626.498887] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9626.504380] LustreError: 321492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9626.886980] Lustre: server umount lustre-MDT0001 complete [ 9636.068695] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 9638.919296] Key type lgssc unregistered [ 9639.231739] LNet: 327797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9639.240782] LNetError: 327797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9639.258692] LNet: Removed LNI 192.168.206.152@tcp [ 9640.099290] Key type .llcrypt unregistered [ 9640.102340] Key type ._llcrypt unregistered [ 9662.300694] Key type ._llcrypt registered [ 9662.303844] Key type .llcrypt registered [ 9662.405911] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 9663.129522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9663.180362] alg: No test for adler32 (adler32-zlib) [ 9664.305307] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 9664.657554] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 9666.336187] Key type lgssc registered [ 9667.419638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9675.043457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9675.090038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9676.644626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9681.814450] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9685.985361] LustreError: 329212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9686.016483] LustreError: 329212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9691.108152] LustreError: 329211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9691.131962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9691.402914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9695.746046] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9706.004197] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9712.925548] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9717.219328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9717.226163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9717.245192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9721.317177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9721.320623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9721.359154] Lustre: Skipped 1 previous similar message [ 9723.174386] Lustre: server umount lustre-MDT0000 complete [ 9726.435537] LustreError: 329212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.471179] LustreError: 329212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9727.012129] LustreError: 329193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558159 with bad export cookie 6490159410703732003 [ 9727.030094] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9727.472758] Lustre: server umount lustre-MDT0001 complete [ 9741.762646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9742.085818] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9742.106575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9742.243879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9746.594788] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9754.492758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9754.567178] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9754.590868] Lustre: 331801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9754.794890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9759.616899] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9767.669615] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9773.421274] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9783.157543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9783.520711] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9783.544693] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9783.553862] LustreError: 333683:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9783.559563] LustreError: 333683:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9783.568860] LustreError: 333683:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9783.643528] Lustre: server umount lustre-OST0000 complete [ 9783.647568] LustreError: 333683:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9794.481085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9794.852442] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9795.098177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9801.857756] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9802.927595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9809.903865] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9819.158248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9819.328298] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9819.425705] Lustre: lustre-OST0001: new disk, initializing [ 9819.432217] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9819.445599] Lustre: 335225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9819.563205] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9825.868325] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 9826.995584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9827.014603] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9827.119251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9833.835799] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9837.544860] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9837.546849] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9837.549778] LustreError: Skipped 1 previous similar message [ 9837.554360] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9837.561770] Lustre: Skipped 1 previous similar message [ 9837.591493] Lustre: Skipped 1 previous similar message [ 9839.478174] Lustre: server umount lustre-OST0001 complete [ 9842.662132] LustreError: 334574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9842.673465] LustreError: 334574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9844.193820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9844.199040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9844.207864] LustreError: Skipped 1 previous similar message [ 9844.222338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9844.235196] Lustre: Skipped 1 previous similar message [ 9847.780929] LustreError: 334150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9847.823168] LustreError: 334150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9849.383127] Lustre: server umount lustre-OST0000 complete [ 9852.907170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9852.926703] Lustre: Skipped 1 previous similar message [ 9852.933173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9852.937339] Lustre: Skipped 2 previous similar messages [ 9857.505495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9863.137093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9863.158641] Lustre: Skipped 4 previous similar messages [ 9867.232356] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9867.566684] Lustre: server umount lustre-MDT0000 complete [ 9868.259256] LustreError: 331811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9868.267327] LustreError: 331811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9872.541397] LustreError: 335224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558304 with bad export cookie 6490159410703732556 [ 9872.548879] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9872.560857] LustreError: 335224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9872.821599] Lustre: server umount lustre-MDT0001 complete [ 9888.300463] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 9892.859601] Key type lgssc unregistered [ 9893.343754] LNet: 337188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9893.356639] LNetError: 337188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9893.379342] LNet: Removed LNI 192.168.206.152@tcp [ 9894.814447] Key type .llcrypt unregistered [ 9894.817719] Key type ._llcrypt unregistered [ 9968.113167] Key type ._llcrypt registered [ 9968.115543] Key type .llcrypt registered [ 9968.250980] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 9982.995721] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 9983.882819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9983.940782] alg: No test for adler32 (adler32-zlib) [ 9985.106916] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [ 9985.368857] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 9987.008353] Key type lgssc registered [ 9988.183916] Lustre: Echo OBD driver; http://www.lustre.org/ [10024.515473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10024.558707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.843592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10025.905659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10026.035697] Lustre: lustre-MDT0000: new disk, initializing [10026.137901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10026.155906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10031.482362] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10041.258249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10041.391306] Lustre: 342122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10041.466474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10041.470662] Lustre: Skipped 1 previous similar message [10041.545561] Lustre: lustre-MDT0001: new disk, initializing [10041.700271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10041.756399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10041.777766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10046.842866] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10056.769075] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10061.840447] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10067.845310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10068.104866] Lustre: lustre-OST0000: new disk, initializing [10068.109158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10068.116433] Lustre: 344106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10068.184376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10074.000732] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10075.196619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10075.207505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10075.328479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10084.282332] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10088.841830] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10089.089933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10093.702161] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10093.900784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10096.098582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10096.101032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10096.132653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10101.234914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10101.238227] Lustre: Skipped 2 previous similar messages [10102.182766] Lustre: server umount lustre-OST0000 complete [10106.338529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.356058] Lustre: Skipped 1 previous similar message [10106.365627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10106.379177] Lustre: Skipped 1 previous similar message [10111.458315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10111.464688] Lustre: Skipped 1 previous similar message [10112.949807] Lustre: server umount lustre-MDT0000 complete [10118.057850] LustreError: 342939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558550 with bad export cookie 8780783316080034114 [10118.063170] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10118.078996] LustreError: 342939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10118.522397] Lustre: server umount lustre-MDT0001 complete [10129.117866] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:02:40 (1787558560) [10138.674180] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:02:50 (1787558570) [10147.881905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10148.301087] LustreError: 346860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10148.337133] LustreError: 346860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10148.413868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10152.411655] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10153.457570] LustreError: 346861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10158.571652] LustreError: 346860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10161.028411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.533353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10166.584972] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10175.792850] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10182.931304] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10191.879187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10192.698782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10202.493975] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10212.894513] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10220.640443] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10222.308967] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10223.911322] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10225.524483] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10227.068899] Lustre: Failing over lustre-MDT0000 [10227.345982] Lustre: server umount lustre-MDT0000 complete [10228.201193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10228.210437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10228.229659] Lustre: Skipped 1 previous similar message [10228.236262] LustreError: 347588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10231.786908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10231.814130] Lustre: Skipped 1 previous similar message [10235.847923] LustreError: 347997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.863453] LustreError: 347997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10244.764515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10244.897403] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10245.072934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10245.133392] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10246.085329] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10249.076876] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10250.233966] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10250.275606] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10250.317129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10257.457549] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10258.839314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10261.955793] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10263.433506] Lustre: Failing over lustre-MDT0000 [10263.621922] Lustre: server umount lustre-MDT0000 complete [10265.577289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10265.580084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10265.580557] LustreError: 346855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10265.580565] LustreError: 346855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10265.587642] Lustre: Skipped 1 previous similar message [10280.933222] Lustre: 340351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787558697/real 1787558697] req@000000001ddf950a x1874390854755328/t0(0) o400->MGC192.168.206.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787558713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10280.970508] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10281.003158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10281.917442] LustreError: 346857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10281.940547] LustreError: 346857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10291.168669] LustreError: 340347:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e54fdd40 x1874390854760064/t0(0) o250->MGC192.168.206.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10291.656916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10291.726925] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10292.167289] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10296.825937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10296.834119] Lustre: Skipped 2 previous similar messages [10296.863160] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10296.868637] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10296.926982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10305.241369] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10306.723048] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10317.283281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10317.284522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.287479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10317.310595] Lustre: Skipped 3 previous similar messages [10318.806644] Lustre: server umount lustre-MDT0000 complete [10322.219231] LustreError: 346840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558754 with bad export cookie 8780783316080036858 [10322.236157] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10322.401648] LustreError: 346856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10322.411059] LustreError: 346856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10322.529681] Lustre: server umount lustre-MDT0001 complete [10328.720114] Lustre: server umount lustre-OST0000 complete [10342.773504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10347.720769] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10356.899601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.324462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10357.334803] Lustre: Skipped 1 previous similar message [10362.265663] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10369.182186] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10374.797878] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10383.089913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10388.658815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10392.014948] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10399.227355] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10409.441446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10409.445449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10409.454901] Lustre: Skipped 1 previous similar message [10409.458830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10409.464164] Lustre: Skipped 2 previous similar messages [10411.029385] Lustre: server umount lustre-OST0000 complete [10418.658071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10421.481709] Lustre: server umount lustre-MDT0000 complete [10423.779397] LustreError: 353200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.797225] LustreError: 353200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10425.131757] LustreError: 353179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558857 with bad export cookie 8780783316080037579 [10425.135672] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10425.144132] LustreError: 353179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10435.857031] Lustre: server umount lustre-MDT0001 complete [10449.961742] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [10452.685600] Key type lgssc unregistered [10452.992148] LNet: 356736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10453.002249] LNetError: 356736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10453.028768] LNet: Removed LNI 192.168.206.152@tcp [10453.877401] Key type .llcrypt unregistered [10453.880706] Key type ._llcrypt unregistered [10476.500210] Key type ._llcrypt registered [10476.502461] Key type .llcrypt registered [10476.589444] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [10477.690973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10477.708545] alg: No test for adler32 (adler32-zlib) [10478.798684] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [10479.091709] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [10480.800975] Key type lgssc registered [10482.162800] Lustre: Echo OBD driver; http://www.lustre.org/ [10490.249130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10490.277866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10492.071952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10496.866969] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10502.633888] LustreError: 358233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10502.670944] LustreError: 358233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10506.410876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10507.111579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10511.618521] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10517.894343] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10522.241288] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10527.618805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10527.866447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10532.982234] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10537.080806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10538.986278] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10551.242686] Lustre: server umount lustre-MDT0000 complete [10552.805216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10552.809757] LustreError: 359484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.816102] Lustre: Skipped 1 previous similar message [10552.836379] LustreError: 359484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10554.178914] LustreError: 358213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558986 with bad export cookie 1138556148795141187 [10554.185139] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10554.189484] LustreError: 358213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10554.430293] Lustre: server umount lustre-MDT0001 complete [10567.800239] Lustre: server umount lustre-OST0000 complete [10578.434583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10578.836605] LustreError: 361944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10578.907211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10582.709095] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10584.035961] LustreError: 361945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10589.159817] LustreError: 361944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10590.724852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10591.120693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10595.578770] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10601.978134] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10606.259611] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10611.822292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10612.065039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10617.168444] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10623.379455] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10624.372850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10639.842904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10639.851685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10639.870238] Lustre: Skipped 3 previous similar messages [10639.883276] Lustre: Skipped 2 previous similar messages [10644.967205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10644.986505] Lustre: Skipped 2 previous similar messages [10645.793765] Lustre: server umount lustre-MDT0000 complete [10649.481310] LustreError: 361926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559081 with bad export cookie 1138556148795142300 [10649.491758] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10649.495559] LustreError: 361926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10649.904135] Lustre: server umount lustre-MDT0001 complete [10664.242510] Lustre: server umount lustre-OST0000 complete [10678.700612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.391609] LustreError: 365853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.514216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10684.540824] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10694.258120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10699.507969] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10707.333512] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10712.425471] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10718.318603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10718.542833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10718.548558] Lustre: Skipped 1 previous similar message [10722.178491] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10724.726268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10726.999539] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10735.077964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10735.079157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10735.080953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10735.088489] Lustre: Skipped 1 previous similar message [10740.200052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10740.209709] Lustre: Skipped 2 previous similar messages [10744.289027] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10744.589583] Lustre: server umount lustre-OST0000 complete [10750.436623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10750.438655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10750.451777] Lustre: Skipped 1 previous similar message [10750.458530] Lustre: Skipped 1 previous similar message [10753.999741] Lustre: server umount lustre-MDT0000 complete [10755.554449] LustreError: 365850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10755.573192] LustreError: 365850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10757.126418] LustreError: 365834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559189 with bad export cookie 1138556148795143322 [10757.135419] LustreError: 365834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10757.137853] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10757.402493] Lustre: server umount lustre-MDT0001 complete [10764.822090] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [10767.089722] Key type lgssc unregistered [10767.270843] LNet: 369390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10767.277563] LNetError: 369390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10767.295279] LNet: Removed LNI 192.168.206.152@tcp [10767.941408] Key type .llcrypt unregistered [10767.942972] Key type ._llcrypt unregistered [10795.264392] Key type ._llcrypt registered [10795.267435] Key type .llcrypt registered [10795.372234] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [10796.329901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10796.366372] alg: No test for adler32 (adler32-zlib) [10797.536767] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [10797.789115] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [10799.464155] Key type lgssc registered [10800.400520] Lustre: Echo OBD driver; http://www.lustre.org/ [10807.286378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10807.306815] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.314954] LustreError: 370830:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10807.369905] LustreError: 370830:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10807.378133] LustreError: 370830:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10807.387577] LustreError: 370830:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10807.399876] LustreError: 370830:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10814.174343] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10814.187392] LustreError: 371228:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10814.217440] LustreError: 371228:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10814.222366] LustreError: 371228:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10814.226514] LustreError: 371228:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10814.230729] LustreError: 371228:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10832.188868] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [10834.766264] Key type lgssc unregistered [10835.046520] LNet: 372236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10835.062172] LNetError: 372236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10835.085147] LNet: Removed LNI 192.168.206.152@tcp [10835.869037] Key type .llcrypt unregistered [10835.872435] Key type ._llcrypt unregistered [10858.547717] Key type ._llcrypt registered [10858.553386] Key type .llcrypt registered [10858.718692] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [10878.215797] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [10879.080715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10879.197000] alg: No test for adler32 (adler32-zlib) [10880.351866] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [10880.648045] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [10882.328158] Key type lgssc registered [10883.476129] Lustre: Echo OBD driver; http://www.lustre.org/ [10912.224373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10912.243387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.485710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10913.510386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10913.588660] Lustre: lustre-MDT0000: new disk, initializing [10913.660851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10913.678924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10917.758619] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10927.614198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.748630] Lustre: 375395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10927.797308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10927.800184] Lustre: Skipped 1 previous similar message [10927.897619] Lustre: lustre-MDT0001: new disk, initializing [10927.992784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10928.032246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10928.045613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10932.697983] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10942.620786] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10946.946830] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10953.110739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10953.448284] Lustre: lustre-OST0000: new disk, initializing [10953.455160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10953.462662] Lustre: 377365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10953.549262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10959.091534] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [10960.965625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10960.973848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10961.044620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10967.368630] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10971.446108] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10971.620318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10976.884381] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10977.144484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10981.871792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10981.874266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10981.900210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10981.906730] Lustre: Skipped 1 previous similar message [10985.460666] Lustre: server umount lustre-OST0000 complete [10989.539854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10989.552214] LustreError: Skipped 1 previous similar message [10989.554436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10989.578957] Lustre: Skipped 1 previous similar message [10989.585789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10992.100440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.104313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10992.117666] Lustre: Skipped 1 previous similar message [10994.908081] Lustre: server umount lustre-MDT0000 complete [10998.240722] LustreError: 375389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559430 with bad export cookie 14390765703906405985 [10998.246156] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10998.255228] LustreError: 375389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10998.540456] Lustre: server umount lustre-MDT0001 complete [11008.076188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11010.542479] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:17:21 (1787559441) [11019.842813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11020.825715] LustreError: 379824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.860877] LustreError: 379824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11020.944102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11026.418313] LustreError: 379825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11028.355242] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11031.525899] LustreError: 379824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11036.644152] LustreError: 379825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.509934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11040.364952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11046.113698] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11057.374828] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11062.566270] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11069.172539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.618782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11075.452240] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11082.204253] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11087.878641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11087.994447] Lustre: lustre-OST0001: new disk, initializing [11087.997623] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11088.002987] Lustre: 382659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11088.081644] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11093.327986] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11093.587222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11093.610259] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11093.687969] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11102.182714] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11109.351128] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11109.365139] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.369385] Lustre: lustre-OST0001: Not available for connect from 192.168.206.52@tcp (stopping) [11109.397642] Lustre: Skipped 1 previous similar message [11109.423302] Lustre: Skipped 2 previous similar messages [11112.043745] Lustre: server umount lustre-OST0001 complete [11114.454880] LustreError: 382842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11114.490435] LustreError: 382842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11124.720207] LustreError: 381649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11124.736574] LustreError: 381649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11129.825697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11129.832726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11129.843048] Lustre: Skipped 1 previous similar message [11129.851880] Lustre: Skipped 1 previous similar message [11134.014830] Lustre: server umount lustre-OST0000 complete [11140.070797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11140.078643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11140.094714] Lustre: Skipped 1 previous similar message [11152.353357] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11152.553610] Lustre: server umount lustre-MDT0000 complete [11155.426221] LustreError: 379819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11155.440904] LustreError: 379819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11155.563707] LustreError: 379805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559587 with bad export cookie 14390765703906406867 [11155.568722] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11155.570411] LustreError: 379805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11162.043589] Lustre: server umount lustre-MDT0001 complete [11174.791522] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [11177.666472] Key type lgssc unregistered [11178.108364] LNet: 384651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11178.117636] LNetError: 384651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11178.148463] LNet: Removed LNI 192.168.206.152@tcp [11178.982170] Key type .llcrypt unregistered [11178.986778] Key type ._llcrypt unregistered [11219.254636] Key type ._llcrypt registered [11219.258623] Key type .llcrypt registered [11219.368238] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [11220.404727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11220.478988] alg: No test for adler32 (adler32-zlib) [11221.451989] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [11221.732793] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [11223.402706] Key type lgssc registered [11224.523838] Lustre: Echo OBD driver; http://www.lustre.org/ [11231.144282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11231.163051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11232.409277] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11232.423600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11232.557976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11236.510995] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11243.772811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11243.882265] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11243.912584] Lustre: 387428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11244.144323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11247.651781] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11252.716634] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11256.496680] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11259.367305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11259.373354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11259.385819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11261.411245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11261.412770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11266.533305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11266.541229] Lustre: Skipped 2 previous similar messages [11271.651593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11271.658832] Lustre: Skipped 1 previous similar message [11272.160568] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11272.343885] Lustre: server umount lustre-MDT0000 complete [11275.530071] LustreError: 388174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559707 with bad export cookie 8965765198935893168 [11275.538727] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11275.546302] LustreError: 388174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11275.843497] Lustre: server umount lustre-MDT0001 complete [11285.052819] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:21:56 (1787559716) [11290.894592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.332678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11295.342444] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11296.743094] LustreError: 389946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.757854] LustreError: 389946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11301.857908] LustreError: 389947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.500962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11302.742773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11305.881516] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11312.413730] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11316.753719] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11323.178305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11323.441327] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11323.707964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11331.777951] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11340.638807] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11347.981988] Lustre: Permanently deactivating lustre-OST0000 [11348.028649] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11357.815439] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11360.245712] LustreError: 392982:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11360.256221] LustreError: 392982:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11360.266053] LustreError: 392982:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11362.957386] LustreError: 393031:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11362.963446] LustreError: 393031:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11371.299574] Lustre: server umount lustre-OST0000 complete [11374.014318] LustreError: 393280:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11374.022352] LustreError: 393280:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11377.648200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11377.649163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11377.673198] Lustre: Skipped 1 previous similar message [11377.695057] Lustre: Skipped 1 previous similar message [11385.234091] Lustre: server umount lustre-MDT0000 complete [11387.874345] LustreError: 390671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11387.907151] LustreError: 390671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11389.872303] LustreError: 389927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559822 with bad export cookie 8965765198935893756 [11389.875641] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11389.898294] LustreError: 389927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11400.500087] Lustre: server umount lustre-MDT0001 complete [11410.255888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.122675] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11418.502105] LustreError: 394439:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11424.843652] Lustre: 394778:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.152@tcp [11427.099235] Lustre: server umount MGS complete [11438.308671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11438.697453] LustreError: 395503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.725710] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11438.736165] Lustre: Skipped 1 previous similar message [11438.846465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11443.243749] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11444.205995] LustreError: 395504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11451.864334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11452.206688] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11456.753260] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11463.862662] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11469.075967] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11470.606341] Lustre: Permanently reactivating lustre-OST0000 [11470.624468] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11470.632230] Lustre: Skipped 2 previous similar messages [11499.494546] LustreError: 397331:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11507.863756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11508.225968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11508.239939] Lustre: Skipped 1 previous similar message [11515.476168] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11523.711180] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11529.700517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11529.704880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11529.706178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11529.711388] Lustre: Skipped 1 previous similar message [11529.716046] Lustre: Skipped 2 previous similar messages [11530.592724] Lustre: server umount lustre-OST0000 complete [11534.305729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11534.315668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11540.724242] Lustre: server umount lustre-MDT0000 complete [11544.309923] LustreError: 395483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559976 with bad export cookie 8965765198935894806 [11544.312928] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11544.333099] LustreError: 395483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11544.661366] Lustre: server umount lustre-MDT0001 complete [11556.998771] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [11561.208709] Key type lgssc unregistered [11561.650321] LNet: 399920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11561.697957] LNetError: 399920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11561.731541] LNet: Removed LNI 192.168.206.152@tcp [11563.085478] Key type .llcrypt unregistered [11563.088128] Key type ._llcrypt unregistered [11589.118983] Key type ._llcrypt registered [11589.121862] Key type .llcrypt registered [11589.258100] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [11609.573499] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [11611.331037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11611.421932] alg: No test for adler32 (adler32-zlib) [11612.582845] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [11612.989210] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [11614.728240] Key type lgssc registered [11615.695238] Lustre: Echo OBD driver; http://www.lustre.org/ [11654.737802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11654.784386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11656.125441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11656.149631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11656.236909] Lustre: lustre-MDT0000: new disk, initializing [11656.314788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11656.331890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11661.252535] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11671.180829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.280503] Lustre: 403081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11671.318708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11671.322574] Lustre: Skipped 1 previous similar message [11671.378656] Lustre: lustre-MDT0001: new disk, initializing [11671.471919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11671.536770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11671.553446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11676.089219] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11685.776385] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11692.766097] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11700.455100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11700.995719] Lustre: lustre-OST0000: new disk, initializing [11701.001152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11701.011073] Lustre: 405049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11701.141143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11708.475572] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11710.170191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11710.191602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11710.365183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11722.418391] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11728.182524] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11728.465088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11734.037448] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11734.391423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11741.161845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11741.167881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11741.176506] LustreError: Skipped 1 previous similar message [11741.180950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11741.227946] Lustre: Skipped 1 previous similar message [11746.282184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11746.284865] Lustre: Skipped 1 previous similar message [11751.393517] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11751.403613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11751.452138] Lustre: Skipped 1 previous similar message [11751.744280] Lustre: server umount lustre-OST0000 complete [11756.515357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11756.566769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11756.578902] Lustre: Skipped 1 previous similar message [11761.635855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11761.654988] Lustre: Skipped 1 previous similar message [11762.834819] Lustre: server umount lustre-MDT0000 complete [11767.609856] LustreError: 403072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560199 with bad export cookie 14117163069957960875 [11767.620618] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11767.625105] LustreError: 403072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11768.031774] Lustre: server umount lustre-MDT0001 complete [11778.403178] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11780.582645] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:30:11 (1787560211) [11788.187678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11788.601060] LustreError: 407513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11788.621820] LustreError: 407513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11788.705611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11793.890886] LustreError: 407514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11795.053222] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11797.987170] LustreError: 407513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11803.110184] LustreError: 407514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11806.062467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11806.750416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11813.435947] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11823.660992] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11832.488451] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11841.734837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11842.280678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11848.180750] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11857.158393] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11859.135646] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11862.102359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11869.153418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11869.166935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11869.175616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11869.214867] Lustre: Skipped 2 previous similar messages [11875.101213] Lustre: server umount lustre-OST0000 complete [11883.490303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11883.495355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11886.156420] Lustre: server umount lustre-MDT0000 complete [11889.645624] LustreError: 407514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11889.685885] LustreError: 407514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11889.804885] LustreError: 407495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560322 with bad export cookie 14117163069957961757 [11889.812888] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11889.821563] LustreError: 407495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11890.354616] Lustre: server umount lustre-MDT0001 complete [11901.312089] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [11904.669944] Key type lgssc unregistered [11905.012419] LNet: 411246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11905.021831] LNetError: 411246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11905.039959] LNet: Removed LNI 192.168.206.152@tcp [11905.947285] Key type .llcrypt unregistered [11905.952676] Key type ._llcrypt unregistered [11931.551484] Key type ._llcrypt registered [11931.553106] Key type .llcrypt registered [11931.672228] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [11932.023287] Key type .llcrypt unregistered [11932.026078] Key type ._llcrypt unregistered [11949.063472] Key type ._llcrypt registered [11949.067468] Key type .llcrypt registered [11949.198426] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [11950.370404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11950.633992] alg: No test for adler32 (adler32-zlib) [11951.634145] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [11951.910726] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [11953.584172] Key type lgssc registered [11954.708093] Lustre: Echo OBD driver; http://www.lustre.org/ [11963.028468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11963.043175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11964.471335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11969.410183] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11974.627606] LustreError: 413469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11974.672785] LustreError: 413469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11978.303870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11978.699482] LustreError: 413470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11978.751558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11983.843699] LustreError: 413469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11983.874985] LustreError: 413469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11984.519974] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [11988.979052] LustreError: 413470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11988.997064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11989.010576] LustreError: 413470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11992.206402] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11994.081821] LustreError: 413469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11994.101683] LustreError: 413469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11998.623307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11998.943616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12003.021802] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12013.131717] Lustre: server umount lustre-OST0000 complete [12019.183280] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12019.194356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12019.214156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12024.803026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12024.810395] Lustre: Skipped 2 previous similar messages [12026.940900] Lustre: server umount lustre-MDT0000 complete [12029.920827] Lustre: 412780:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787560446/real 1787560446] req@0000000019a6971b x1874392917239168/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787560462 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12029.923909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12029.926771] LustreError: 413464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12029.937919] Lustre: 412780:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12029.948716] Lustre: Skipped 1 previous similar message [12030.287936] LustreError: 413448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560462 with bad export cookie 3340438777378789144 [12030.304190] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12030.578558] Lustre: server umount lustre-MDT0001 complete [12040.665824] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [12043.169643] Key type lgssc unregistered [12043.405553] LNet: 416616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12043.413621] LNetError: 416616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12043.425954] LNet: Removed LNI 192.168.206.152@tcp [12044.071875] Key type .llcrypt unregistered [12044.074867] Key type ._llcrypt unregistered [12063.758700] Key type ._llcrypt registered [12063.762517] Key type .llcrypt registered [12063.907512] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [12064.839463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12064.905976] alg: No test for adler32 (adler32-zlib) [12066.257921] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [12066.597847] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [12068.332440] Key type lgssc registered [12069.489293] Lustre: Echo OBD driver; http://www.lustre.org/ [12076.828108] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12076.863364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12092.448437] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12097.568284] LustreError: 417984:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.152@tcp: failed processing log, type 4: rc = -110 [12124.256282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12130.882906] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12138.103363] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12144.823234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12145.378556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12145.466436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12149.223861] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12150.759980] LustreError: 418010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12150.779304] LustreError: 418010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12155.878818] LustreError: 418009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.119067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12157.605652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12161.944990] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12170.134441] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12177.191919] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12186.596222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12186.601904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12186.611798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12188.642750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12188.654622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12188.676867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12190.690818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12190.708032] Lustre: Skipped 1 previous similar message [12192.364282] Lustre: server umount lustre-OST0000 complete [12198.884494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12198.896451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12198.913266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12200.931862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.561060] Lustre: server umount lustre-MDT0000 complete [12206.051331] LustreError: 419025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12206.063533] LustreError: 419025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12206.277394] LustreError: 417990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560638 with bad export cookie 3289562710471117723 [12206.281497] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12206.293056] LustreError: 417990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12206.530690] Lustre: server umount lustre-MDT0001 complete [12217.724910] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [12220.475336] Key type lgssc unregistered [12220.865513] LNet: 421556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12220.884752] LNetError: 421556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12220.911758] LNet: Removed LNI 192.168.206.152@tcp [12221.683568] Key type .llcrypt unregistered [12221.685491] Key type ._llcrypt unregistered [12244.976755] Key type ._llcrypt registered [12244.979353] Key type .llcrypt registered [12245.095904] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [12246.225800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12246.322437] alg: No test for adler32 (adler32-zlib) [12247.478956] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [12247.720860] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [12249.504151] Key type lgssc registered [12250.422542] Lustre: Echo OBD driver; http://www.lustre.org/ [12258.487464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12258.505875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12260.279209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12266.502589] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12270.563596] LustreError: 422976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12270.587720] LustreError: 422976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12275.552803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12275.685671] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12276.011061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12280.453392] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12288.363653] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12293.977608] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12299.973886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12300.425518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12305.580565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12305.582745] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12311.470538] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12316.130327] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12316.142993] Lustre: Skipped 1 previous similar message [12316.143057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12316.156507] Lustre: Skipped 2 previous similar messages [12319.190061] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.52@tcp (stopping) [12321.253827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12321.258199] Lustre: Skipped 2 previous similar messages [12321.993033] Lustre: server umount lustre-MDT0000 complete [12323.956061] LustreError: 423709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.52@tcp arrived at 1787560756 with bad export cookie 5602629421398583223 [12323.968622] LustreError: 423709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12325.392436] LustreError: 422956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560757 with bad export cookie 5602629421398582327 [12325.401097] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12325.408481] LustreError: 422956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12326.368975] LustreError: 422975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.371919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12326.385969] LustreError: 422975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12326.393602] Lustre: Skipped 1 previous similar message [12326.395037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12330.401939] LustreError: 424217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12330.420100] LustreError: 424217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12331.655390] Lustre: server umount lustre-OST0000 complete [12335.523681] LustreError: 422975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12335.538913] LustreError: 422975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12337.648873] Lustre: server umount lustre-MDT0001 complete [12348.739961] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [12351.623850] Key type lgssc unregistered [12351.918837] LNet: 426661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12351.934873] LNetError: 426661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12351.959900] LNet: Removed LNI 192.168.206.152@tcp [12353.006453] Key type .llcrypt unregistered [12353.009716] Key type ._llcrypt unregistered [12379.075386] Key type ._llcrypt registered [12379.076833] Key type .llcrypt registered [12379.210148] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [12379.999710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12380.116265] alg: No test for adler32 (adler32-zlib) [12381.301158] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [12381.605916] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [12383.384462] Key type lgssc registered [12384.411530] Lustre: Echo OBD driver; http://www.lustre.org/ [12393.212580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12393.271197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12394.884410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12400.015164] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12405.220701] LustreError: 428079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12405.239825] LustreError: 428079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12408.972879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12409.290399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12414.395208] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12422.652807] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12429.186868] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12437.212091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12437.631559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12440.690538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12443.738354] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12452.218055] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12459.498495] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12459.529774] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12459.538177] Lustre: Skipped 1 previous similar message [12463.502404] Lustre: server umount lustre-MDT0001 complete [12464.611464] LustreError: 428080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12464.633414] LustreError: 428080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12469.732086] LustreError: 428079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.754669] LustreError: 428079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12471.780251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12471.789728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12471.797416] Lustre: Skipped 1 previous similar message [12471.801268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12471.805416] Lustre: Skipped 1 previous similar message [12474.856978] LustreError: 428821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12474.859673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12476.512430] Lustre: server umount lustre-OST0000 complete [12479.988966] Lustre: server umount lustre-MDT0000 complete [12491.441790] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [12493.909368] Key type lgssc unregistered [12494.214864] LNet: 431766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12494.234945] LNetError: 431766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12494.254846] LNet: Removed LNI 192.168.206.152@tcp [12495.077643] Key type .llcrypt unregistered [12495.086229] Key type ._llcrypt unregistered [12520.919500] Key type ._llcrypt registered [12520.923133] Key type .llcrypt registered [12521.019859] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [12540.602248] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [12542.066969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12542.190211] alg: No test for adler32 (adler32-zlib) [12543.556365] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [12543.933111] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [12545.672466] Key type lgssc registered [12547.189639] Lustre: Echo OBD driver; http://www.lustre.org/ [12583.045747] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [12598.847877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12598.897191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12600.267934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12600.305394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12600.464645] Lustre: lustre-MDT0000: new disk, initializing [12600.602672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12600.622666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12605.116687] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12621.830795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12622.042072] Lustre: lustre-OST0000: new disk, initializing [12622.044987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12622.047729] Lustre: Skipped 1 previous similar message [12622.051483] Lustre: 436804:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12622.106193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12628.764959] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12632.199467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12632.209919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12632.241756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12638.948422] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12653.662361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12653.837781] Lustre: 435785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12653.985284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12654.134882] Lustre: lustre-MDT0001: new disk, initializing [12654.270561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12654.329626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12654.352359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12659.802675] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12670.282583] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12670.475761] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12676.036222] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12676.460617] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12684.771190] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12684.780288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12684.821547] Lustre: Skipped 1 previous similar message [12684.822990] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12684.828984] Lustre: Skipped 1 previous similar message [12685.784960] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.52@tcp (stopping) [12687.492716] Lustre: server umount lustre-MDT0001 complete [12695.011229] LustreError: 435793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12695.024155] LustreError: 435793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12700.131184] LustreError: 438380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12700.151106] LustreError: 438380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12705.252506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12705.254936] LustreError: 435796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12705.276377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12710.372925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12710.373071] LustreError: 438380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12710.922800] Lustre: server umount lustre-OST0000 complete [12721.032769] Lustre: server umount lustre-MDT0000 complete [12733.979459] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [12736.514196] Key type lgssc unregistered [12736.738608] LNet: 440561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12736.745027] LNetError: 440561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12736.755710] LNet: Removed LNI 192.168.206.152@tcp [12737.542746] Key type .llcrypt unregistered [12737.545239] Key type ._llcrypt unregistered [12774.108355] Key type ._llcrypt registered [12774.110377] Key type .llcrypt registered [12774.230770] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [12774.556359] Key type .llcrypt unregistered [12774.559191] Key type ._llcrypt unregistered [12790.520158] Key type ._llcrypt registered [12790.522621] Key type .llcrypt registered [12790.622880] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [12791.947421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12791.964051] alg: No test for adler32 (adler32-zlib) [12793.051775] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [12793.267527] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [12794.936238] Key type lgssc registered [12796.198306] Lustre: Echo OBD driver; http://www.lustre.org/ [12822.415800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12822.446487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12823.652711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12823.679049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12823.736733] Lustre: lustre-MDT0000: new disk, initializing [12823.795489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12823.818859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12828.678711] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12839.161143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12839.365869] Lustre: 443944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12839.421614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12839.425848] Lustre: Skipped 1 previous similar message [12839.483629] Lustre: lustre-MDT0001: new disk, initializing [12839.561425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12839.608611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12839.617244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12843.727614] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12853.252817] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12860.119836] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12868.010943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12868.363879] Lustre: lustre-OST0000: new disk, initializing [12868.375204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12868.383982] Lustre: 445912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12868.457742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12875.748530] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12876.870986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12876.898348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12877.038810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12887.527848] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12896.232312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12896.241592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12896.269081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12897.763366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12897.765339] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12897.781960] Lustre: Skipped 1 previous similar message [12897.820540] Lustre: Skipped 1 previous similar message [12901.818355] Lustre: server umount lustre-MDT0000 complete [12905.462873] LustreError: 444738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561337 with bad export cookie 17802307400897984505 [12905.471318] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12905.479770] LustreError: 444738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12905.875326] Lustre: server umount lustre-MDT0001 complete [12920.964756] Lustre: server umount lustre-OST0000 complete [12932.662633] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:49:24 (1787561364) [12939.960245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12940.405979] LustreError: 447943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12940.430214] LustreError: 447943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12940.472209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12944.414871] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12945.891914] LustreError: 447944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12949.988821] LustreError: 447943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12953.530108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12954.086128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12959.468508] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [12968.284237] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12974.666973] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12994.934714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12995.185769] Lustre: Found index 0 for lustre-OST0000, updating log [12995.381110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13001.549830] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [13008.131923] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13012.828983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13018.089910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13018.102565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13018.107598] Lustre: Skipped 1 previous similar message [13020.339064] Lustre: server umount lustre-OST0000 complete [13036.002327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13036.012860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13036.032320] Lustre: Skipped 1 previous similar message [13036.051481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13036.058186] Lustre: Skipped 3 previous similar messages [13040.611412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13040.615168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13040.628619] Lustre: Skipped 1 previous similar message [13050.338882] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13050.551103] Lustre: server umount lustre-MDT0000 complete [13051.874624] LustreError: 447938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.887596] LustreError: 447938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13054.056663] LustreError: 447925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561486 with bad export cookie 17802307400897985898 [13054.063902] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13054.068377] LustreError: 447925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13054.459551] Lustre: server umount lustre-MDT0001 complete [13056.480161] LNet: 1 local NIs in recovery (showing 1): 192.168.206.152@tcp [13064.148030] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:51:35 (1787561495) [13686.740891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13687.230644] LustreError: 454365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13687.307990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13691.668573] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [13692.392172] LustreError: 454366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13697.505378] LustreError: 454365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13699.609889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13699.890859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13703.829695] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [13711.253350] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13717.704708] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13738.227592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13738.767699] Lustre: Found index 0 for lustre-OST0000, updating log [13739.081157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13746.706604] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [13747.747955] LNet: 1 local NIs in recovery (showing 1): 192.168.206.152@tcp [13755.849353] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13758.534610] Lustre: server umount lustre-OST0000 complete [13763.555838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13763.564830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13763.581033] Lustre: Skipped 1 previous similar message [13763.598539] Lustre: Skipped 5 previous similar messages [13764.660880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13764.670848] Lustre: Skipped 1 previous similar message [13766.692634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13766.703867] Lustre: Skipped 3 previous similar messages [13769.038105] Lustre: server umount lustre-MDT0000 complete [13769.770690] LustreError: 454366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13769.781919] LustreError: 454366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13773.152732] LustreError: 456230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562205 with bad export cookie 17802307400897986976 [13773.154899] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13773.161903] LustreError: 456230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13773.724352] Lustre: server umount lustre-MDT0001 complete [13804.279265] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [13813.553445] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [13849.234663] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:04:40 (1787562280) [13868.054308] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [13870.943632] Key type lgssc unregistered [13871.237620] LNet: 460995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13871.247124] LNetError: 460995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14001.322302] LNet: Removed LNI 192.168.206.152@tcp [14002.661516] Key type .llcrypt unregistered [14002.664916] Key type ._llcrypt unregistered [14020.432550] Key type ._llcrypt registered [14020.436685] Key type .llcrypt registered [14020.552816] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [14021.780687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14021.809956] alg: No test for adler32 (adler32-zlib) [14022.825055] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [14023.021654] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [14024.664191] Key type lgssc registered [14025.799911] Lustre: Echo OBD driver; http://www.lustre.org/ [14029.938677] LNet: Added LNI 192.168.206.152@tcp100 [8/256/0/180] [14030.743154] LNet: Added LNI 192.168.206.152@tcp101 [8/256/0/180] [14031.642868] LNet: Added LNI 192.168.206.152@tcp102 [8/256/0/180] [14032.362686] LNet: Added LNI 192.168.206.152@tcp103 [8/256/0/180] [14033.152686] LNet: Added LNI 192.168.206.152@tcp104 [8/256/0/180] [14034.059526] LNet: Added LNI 192.168.206.152@tcp105 [8/256/0/180] [14034.869692] LNet: Added LNI 192.168.206.152@tcp106 [8/256/0/180] [14035.870634] LNet: Added LNI 192.168.206.152@tcp107 [8/256/0/180] [14036.748261] LNet: Added LNI 192.168.206.152@tcp108 [8/256/0/180] [14037.682337] LNet: Added LNI 192.168.206.152@tcp109 [8/256/0/180] [14038.639331] LNet: Added LNI 192.168.206.152@tcp110 [8/256/0/180] [14039.859959] LNet: Added LNI 192.168.206.152@tcp111 [8/256/0/180] [14040.946579] LNet: Added LNI 192.168.206.152@tcp112 [8/256/0/180] [14041.917521] LNet: Added LNI 192.168.206.152@tcp113 [8/256/0/180] [14042.928936] LNet: Added LNI 192.168.206.152@tcp114 [8/256/0/180] [14044.025149] LNet: Added LNI 192.168.206.152@tcp115 [8/256/0/180] [14045.143650] LNet: Added LNI 192.168.206.152@tcp116 [8/256/0/180] [14045.894435] LNet: Added LNI 192.168.206.152@tcp117 [8/256/0/180] [14046.925524] LNet: Added LNI 192.168.206.152@tcp118 [8/256/0/180] [14047.983531] LNet: Added LNI 192.168.206.152@tcp119 [8/256/0/180] [14049.006104] LNet: Added LNI 192.168.206.152@tcp120 [8/256/0/180] [14050.025929] LNet: Added LNI 192.168.206.152@tcp121 [8/256/0/180] [14050.811361] LNet: Added LNI 192.168.206.152@tcp122 [8/256/0/180] [14051.813275] LNet: Added LNI 192.168.206.152@tcp123 [8/256/0/180] [14052.787810] LNet: Added LNI 192.168.206.152@tcp124 [8/256/0/180] [14053.661850] LNet: Added LNI 192.168.206.152@tcp125 [8/256/0/180] [14054.793768] LNet: Added LNI 192.168.206.152@tcp126 [8/256/0/180] [14055.750761] LNet: Added LNI 192.168.206.152@tcp127 [8/256/0/180] [14056.848839] LNet: Added LNI 192.168.206.152@tcp128 [8/256/0/180] [14058.166904] LNet: Added LNI 192.168.206.152@tcp129 [8/256/0/180] [14059.200592] LNet: Added LNI 192.168.206.152@tcp130 [8/256/0/180] [14060.232581] LNet: Added LNI 192.168.206.152@tcp131 [8/256/0/180] [14061.364815] LNet: Added LNI 192.168.206.152@tcp132 [8/256/0/180] [14062.265414] LNet: Added LNI 192.168.206.152@tcp133 [8/256/0/180] [14063.330808] LNet: Added LNI 192.168.206.152@tcp134 [8/256/0/180] [14064.239802] LNet: Added LNI 192.168.206.152@tcp135 [8/256/0/180] [14072.777933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14072.853838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14074.028129] Lustre: 464031:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.152@tcp: can't add NID 192.168.206.152@tcp130: rc = -75 [14074.106181] LNetError: 464056:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [14074.119690] LNetError: Protocol error connecting to 192.168.206.152@tcp120 at host 192.168.206.152:988: is it running a compatible version of Lustre? [14074.138816] LNetError: Skipped 7 previous similar messages [14074.208167] LNet: 5 local NIs in recovery (showing 5): 192.168.206.152@tcp125, 192.168.206.152@tcp124, 192.168.206.152@tcp128, 192.168.206.152@tcp129, 192.168.206.152@tcp120 [14074.244545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14074.262137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14074.317652] Lustre: lustre-MDT0000: new disk, initializing [14074.371242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14074.381980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14075.173795] LNetError: 464058:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [14075.194426] LNetError: 464058:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14075.221326] LNetError: Protocol error connecting to 192.168.206.152@tcp108 at host 192.168.206.152:988: is it running a compatible version of Lustre? [14075.259465] LNetError: Skipped 2 previous similar messages [14077.345681] LNetError: 464056:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [14077.384825] LNetError: 464056:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14077.420716] LNetError: Protocol error connecting to 192.168.206.152@tcp125 at host 192.168.206.152:988: is it running a compatible version of Lustre? [14080.127694] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14081.442952] LNetError: 461486:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [14081.468946] LNetError: Protocol error connecting to 192.168.206.152@tcp125 at host 192.168.206.152:988: is it running a compatible version of Lustre? [14089.634076] LNetError: 464058:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [14089.646070] LNetError: Protocol error connecting to 192.168.206.152@tcp125 at host 192.168.206.152:988: is it running a compatible version of Lustre? [14091.154564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14091.339886] Lustre: 464046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14091.477617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14091.480562] Lustre: 464850:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.152@tcp: can't add NID 192.168.206.152@tcp130: rc = -75 [14091.484892] Lustre: Skipped 1 previous similar message [14091.509535] Lustre: 464850:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14091.537497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14091.630107] Lustre: lustre-MDT0001: new disk, initializing [14091.830209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14091.877073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14091.896947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14096.636695] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14104.993602] LNetError: 464051:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [14105.011938] LNetError: Protocol error connecting to 192.168.206.152@tcp119 at host 192.168.206.152:988: is it running a compatible version of Lustre? [14107.198604] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14113.253528] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14120.956142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14121.424148] Lustre: lustre-OST0000: new disk, initializing [14121.433414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14121.447632] Lustre: 466024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14121.618588] Lustre: 466029:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.152@tcp: can't add NID 192.168.206.152@tcp130: rc = -75 [14121.626746] Lustre: 466029:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14121.700493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14123.432189] LNetError: 464051:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [14123.456862] LNetError: 464051:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14123.467622] LNetError: Protocol error connecting to 192.168.206.152@tcp123 at host 192.168.206.152:988: is it running a compatible version of Lustre? [14123.478501] LNetError: Skipped 9 previous similar messages [14127.650153] Lustre: 466310:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.152@tcp: can't add NID 192.168.206.152@tcp130: rc = -75 [14127.658134] Lustre: 466310:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14127.724707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14127.763105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14127.997716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14129.087710] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14134.753904] LNet: 1 local NIs in recovery (showing 1): 192.168.206.152@tcp125 [14139.056794] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14148.578237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14148.585470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14148.598606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14148.608678] Lustre: Skipped 1 previous similar message [14152.494073] Lustre: server umount lustre-OST0000 complete [14158.311547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14158.322081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14158.338544] Lustre: Skipped 1 previous similar message [14158.344125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14162.327027] Lustre: server umount lustre-MDT0000 complete [14165.409339] LNetError: 464058:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [14165.431453] LNetError: 464058:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [14165.436878] LNetError: Protocol error connecting to 192.168.206.152@tcp108 at host 192.168.206.152:988: is it running a compatible version of Lustre? [14165.448172] LNetError: Skipped 18 previous similar messages [14165.831139] LustreError: 464039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562598 with bad export cookie 3084039197119507363 [14165.836470] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14165.841796] LustreError: 464039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14166.198991] Lustre: server umount lustre-MDT0001 complete [14176.287547] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [14178.642211] Key type lgssc unregistered [14179.014488] LNet: 467939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14179.022971] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14179.028239] LNet: Removed LNI 192.168.206.152@tcp [14180.064403] LNet: Removed LNI 192.168.206.152@tcp100 [14180.071567] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14180.082322] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14181.152215] LNet: Removed LNI 192.168.206.152@tcp101 [14181.176671] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14181.186287] LNet: Removed LNI 192.168.206.152@tcp102 [14182.240219] LNet: Removed LNI 192.168.206.152@tcp103 [14183.264169] LNet: Removed LNI 192.168.206.152@tcp104 [14183.276766] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14183.288745] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14183.310668] LNet: Removed LNI 192.168.206.152@tcp105 [14184.352134] LNet: Removed LNI 192.168.206.152@tcp106 [14185.377159] LNet: Removed LNI 192.168.206.152@tcp107 [14186.464121] LNet: Removed LNI 192.168.206.152@tcp108 [14187.488226] LNet: Removed LNI 192.168.206.152@tcp109 [14187.492611] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14187.501969] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14188.513675] LNet: Removed LNI 192.168.206.152@tcp110 [14189.536187] LNet: Removed LNI 192.168.206.152@tcp111 [14190.563262] LNet: Removed LNI 192.168.206.152@tcp112 [14191.587226] LNet: Removed LNI 192.168.206.152@tcp113 [14192.608389] LNet: Removed LNI 192.168.206.152@tcp114 [14193.697388] LNet: Removed LNI 192.168.206.152@tcp115 [14194.720182] LNet: Removed LNI 192.168.206.152@tcp116 [14195.744182] LNet: Removed LNI 192.168.206.152@tcp117 [14195.748647] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14195.769458] LNetError: 467939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [14195.792281] LNet: Removed LNI 192.168.206.152@tcp118 [14196.833318] LNet: Removed LNI 192.168.206.152@tcp119 [14197.856812] LNet: Removed LNI 192.168.206.152@tcp120 [14198.881157] LNet: Removed LNI 192.168.206.152@tcp121 [14199.904168] LNet: Removed LNI 192.168.206.152@tcp122 [14200.928153] LNet: Removed LNI 192.168.206.152@tcp123 [14201.952312] LNet: Removed LNI 192.168.206.152@tcp124 [14201.955172] LNet: Removed LNI 192.168.206.152@tcp125 [14202.977975] LNet: Removed LNI 192.168.206.152@tcp126 [14204.000367] LNet: Removed LNI 192.168.206.152@tcp127 [14205.025626] LNet: Removed LNI 192.168.206.152@tcp128 [14206.049160] LNet: Removed LNI 192.168.206.152@tcp129 [14206.061794] LNet: Removed LNI 192.168.206.152@tcp130 [14206.067190] LNet: Removed LNI 192.168.206.152@tcp131 [14206.076198] LNet: Removed LNI 192.168.206.152@tcp132 [14206.084363] LNet: Removed LNI 192.168.206.152@tcp133 [14206.088326] LNet: Removed LNI 192.168.206.152@tcp134 [14206.099518] LNet: Removed LNI 192.168.206.152@tcp135 [14207.052234] Key type .llcrypt unregistered [14207.054350] Key type ._llcrypt unregistered [14230.039466] Key type ._llcrypt registered [14230.043138] Key type .llcrypt registered [14230.171440] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [14248.911448] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [14251.139374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14251.181753] alg: No test for adler32 (adler32-zlib) [14252.411478] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [14252.736480] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [14254.545463] Key type lgssc registered [14256.130355] Lustre: Echo OBD driver; http://www.lustre.org/ [14291.358848] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:12:02 (1787562722) [14310.016681] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [14313.142059] Key type lgssc unregistered [14313.520825] LNet: 472031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14313.540466] LNetError: 472031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14313.562916] LNet: Removed LNI 192.168.206.152@tcp [14314.513605] Key type .llcrypt unregistered [14314.517469] Key type ._llcrypt unregistered [14332.464684] Key type ._llcrypt registered [14332.466582] Key type .llcrypt registered [14332.549436] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [14334.188818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14334.238453] alg: No test for adler32 (adler32-zlib) [14335.410270] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [14335.678363] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [14337.432183] Key type lgssc registered [14338.670638] Lustre: Echo OBD driver; http://www.lustre.org/ [14349.652293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14349.689429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14351.024125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14351.070604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14351.185965] Lustre: lustre-MDT0000: new disk, initializing [14351.270514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14351.289326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14355.958155] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14366.315319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14366.424935] Lustre: 473247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14366.477942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14366.481425] Lustre: Skipped 1 previous similar message [14366.549248] Lustre: lustre-MDT0001: new disk, initializing [14366.647345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14366.697059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14366.722944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14371.635225] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14381.728334] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14387.207243] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14394.299238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14394.703608] Lustre: lustre-OST0000: new disk, initializing [14394.715861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14394.731752] Lustre: 475215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14394.811617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14400.984801] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14401.603261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14401.616118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14401.667696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14410.727312] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14415.088628] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing lnet_if_list [14420.770727] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14420.811196] LNet: Added LNI 192.168.206.152@tcp10 [8/256/0/180] [14427.339966] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14427.385494] LNet: Added LNI 192.168.206.152@tcp11 [8/256/0/180] [14431.949820] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14431.991745] LNet: Added LNI 192.168.206.152@tcp12 [8/256/0/180] [14436.440397] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14436.483153] LNet: Added LNI 192.168.206.152@tcp13 [8/256/0/180] [14441.390874] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14441.421252] LNet: Added LNI 192.168.206.152@tcp14 [8/256/0/180] [14445.902893] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14445.935094] LNet: Added LNI 192.168.206.152@tcp15 [8/256/0/180] [14451.048424] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14451.078521] LNet: Added LNI 192.168.206.152@tcp16 [8/256/0/180] [14455.967659] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14456.018866] LNet: Added LNI 192.168.206.152@tcp17 [8/256/0/180] [14461.753783] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14461.779346] LNet: Added LNI 192.168.206.152@tcp18 [8/256/0/180] [14466.660410] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14466.681306] LNet: Added LNI 192.168.206.152@tcp19 [8/256/0/180] [14472.586162] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14472.616878] LNet: Added LNI 192.168.206.152@tcp20 [8/256/0/180] [14477.830457] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14477.880027] LNet: Added LNI 192.168.206.152@tcp21 [8/256/0/180] [14483.640236] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14483.678572] LNet: Added LNI 192.168.206.152@tcp22 [8/256/0/180] [14489.543993] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14489.588590] LNet: Added LNI 192.168.206.152@tcp23 [8/256/0/180] [14496.318507] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14496.353875] LNet: Added LNI 192.168.206.152@tcp24 [8/256/0/180] [14502.152861] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14502.194257] LNet: Added LNI 192.168.206.152@tcp25 [8/256/0/180] [14507.925584] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14507.958921] LNet: Added LNI 192.168.206.152@tcp26 [8/256/0/180] [14513.459995] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14513.497719] LNet: Added LNI 192.168.206.152@tcp27 [8/256/0/180] [14519.787937] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14519.822670] LNet: Added LNI 192.168.206.152@tcp28 [8/256/0/180] [14527.277327] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14527.324261] LNet: Added LNI 192.168.206.152@tcp29 [8/256/0/180] [14533.399981] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14533.432560] LNet: Added LNI 192.168.206.152@tcp30 [8/256/0/180] [14538.986044] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14539.024359] LNet: Added LNI 192.168.206.152@tcp31 [8/256/0/180] [14543.829903] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14543.851961] LNet: Added LNI 192.168.206.152@tcp32 [8/256/0/180] [14552.058068] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14552.098890] LNet: Added LNI 192.168.206.152@tcp33 [8/256/0/180] [14563.274709] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14563.301589] LNet: Added LNI 192.168.206.152@tcp34 [8/256/0/180] [14570.778617] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14570.831729] LNet: Added LNI 192.168.206.152@tcp35 [8/256/0/180] [14578.148118] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14578.177851] LNet: Added LNI 192.168.206.152@tcp36 [8/256/0/180] [14585.740578] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14585.770829] LNet: Added LNI 192.168.206.152@tcp37 [8/256/0/180] [14593.738433] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14593.764775] LNet: Added LNI 192.168.206.152@tcp38 [8/256/0/180] [14600.976317] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14600.996964] LNet: Added LNI 192.168.206.152@tcp39 [8/256/0/180] [14608.449105] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14608.471463] LNet: Added LNI 192.168.206.152@tcp40 [8/256/0/180] [14615.152963] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14615.185421] LNet: Added LNI 192.168.206.152@tcp41 [8/256/0/180] [14622.974302] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14622.995074] LNet: Added LNI 192.168.206.152@tcp42 [8/256/0/180] [14631.741980] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14631.778085] LNet: Added LNI 192.168.206.152@tcp43 [8/256/0/180] [14640.098535] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14640.126199] LNet: Added LNI 192.168.206.152@tcp44 [8/256/0/180] [14648.673321] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14648.696871] LNet: Added LNI 192.168.206.152@tcp45 [8/256/0/180] [14656.534390] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14656.564465] LNet: Added LNI 192.168.206.152@tcp46 [8/256/0/180] [14664.289261] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14664.315443] LNet: Added LNI 192.168.206.152@tcp47 [8/256/0/180] [14672.180692] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14672.212803] LNet: Added LNI 192.168.206.152@tcp48 [8/256/0/180] [14680.539256] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14680.579272] LNet: Added LNI 192.168.206.152@tcp49 [8/256/0/180] [14689.000085] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14689.029205] LNet: Added LNI 192.168.206.152@tcp50 [8/256/0/180] [14709.222086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14709.226813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14709.240384] Lustre: Skipped 1 previous similar message [14712.764771] Lustre: server umount lustre-MDT0000 complete [14716.195572] LustreError: 473237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563148 with bad export cookie 13024556886134416410 [14716.201361] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14716.205167] LustreError: 473237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14716.771410] Lustre: server umount lustre-MDT0001 complete [14734.821363] Lustre: 472518:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787563151/real 1787563151] req@000000009ac41af4 x1874395416141568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787563167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14734.864940] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14734.896055] Lustre: Skipped 1 previous similar message [14736.853543] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [14736.992153] Lustre: 472518:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787563153/real 1787563153] req@000000004a2dcae0 x1874395416141696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787563169 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14739.944674] Lustre: 472520:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787563156/real 1787563156] req@00000000f29c8af2 x1874395416141824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787563172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14741.985081] Lustre: 472519:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787563158/real 1787563158] req@000000009093c00a x1874395416142080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787563174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14745.534283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14748.128162] Lustre: 472517:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787563164/real 1787563164] req@00000000d3b4cd9c x1874395416142208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787563180 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14761.897053] LustreError: 475208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14761.915209] LustreError: 475208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14761.950422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14764.001637] LustreError: 475209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14764.007821] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14766.052221] LustreError: 475208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14766.579278] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14771.174244] LustreError: 475209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14775.024704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14775.469846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14779.802967] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14780.590545] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14786.524413] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14792.009339] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14811.108820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14811.120251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14811.143285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14811.151797] Lustre: Skipped 2 previous similar messages [14815.681555] Lustre: server umount lustre-OST0000 complete [14821.345814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14821.357091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14821.375055] Lustre: Skipped 1 previous similar message [14821.393715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14821.397532] Lustre: Skipped 1 previous similar message [14825.819964] Lustre: server umount lustre-MDT0000 complete [14826.977859] LustreError: 499464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14827.001800] LustreError: 499464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14829.459195] LustreError: 476427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563261 with bad export cookie 13024556886134419035 [14829.463723] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14829.474468] LustreError: 476427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14829.752802] Lustre: server umount lustre-MDT0001 complete [14843.200290] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [14847.395196] Key type lgssc unregistered [14847.973215] LNet: 502244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14847.990668] LNetError: 502244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14849.056376] LNet: Removed LNI 192.168.206.152@tcp [14849.060727] LNetError: 502244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14849.076880] LNet: Removed LNI 192.168.206.152@tcp10 [14849.084568] LNet: Removed LNI 192.168.206.152@tcp11 [14849.092158] LNet: Removed LNI 192.168.206.152@tcp12 [14849.107429] LNet: Removed LNI 192.168.206.152@tcp13 [14849.116359] LNet: Removed LNI 192.168.206.152@tcp14 [14849.120309] LNet: Removed LNI 192.168.206.152@tcp15 [14849.128156] LNet: Removed LNI 192.168.206.152@tcp16 [14849.136618] LNet: Removed LNI 192.168.206.152@tcp17 [14849.141141] LNet: Removed LNI 192.168.206.152@tcp18 [14849.153755] LNet: Removed LNI 192.168.206.152@tcp19 [14849.169699] LNet: Removed LNI 192.168.206.152@tcp20 [14849.184659] LNet: Removed LNI 192.168.206.152@tcp21 [14849.192516] LNet: Removed LNI 192.168.206.152@tcp22 [14849.201240] LNet: Removed LNI 192.168.206.152@tcp23 [14849.216406] LNet: Removed LNI 192.168.206.152@tcp24 [14849.226835] LNet: Removed LNI 192.168.206.152@tcp25 [14849.230603] LNet: Removed LNI 192.168.206.152@tcp26 [14849.237155] LNet: Removed LNI 192.168.206.152@tcp27 [14849.244662] LNet: Removed LNI 192.168.206.152@tcp28 [14849.254535] LNet: Removed LNI 192.168.206.152@tcp29 [14849.266202] LNet: Removed LNI 192.168.206.152@tcp30 [14849.269890] LNet: Removed LNI 192.168.206.152@tcp31 [14849.274964] LNet: Removed LNI 192.168.206.152@tcp32 [14849.281168] LNet: Removed LNI 192.168.206.152@tcp33 [14849.283404] LNet: Removed LNI 192.168.206.152@tcp34 [14849.287904] LNet: Removed LNI 192.168.206.152@tcp35 [14849.291289] LNet: Removed LNI 192.168.206.152@tcp36 [14849.299825] LNet: Removed LNI 192.168.206.152@tcp37 [14849.304305] LNet: Removed LNI 192.168.206.152@tcp38 [14849.312767] LNet: Removed LNI 192.168.206.152@tcp39 [14849.322963] LNet: Removed LNI 192.168.206.152@tcp40 [14849.328438] LNet: Removed LNI 192.168.206.152@tcp41 [14849.345162] LNet: Removed LNI 192.168.206.152@tcp42 [14849.355173] LNet: Removed LNI 192.168.206.152@tcp43 [14849.359288] LNet: Removed LNI 192.168.206.152@tcp44 [14849.364689] LNet: Removed LNI 192.168.206.152@tcp45 [14849.373094] LNet: Removed LNI 192.168.206.152@tcp46 [14849.379142] LNet: Removed LNI 192.168.206.152@tcp47 [14849.383041] LNet: Removed LNI 192.168.206.152@tcp48 [14849.387245] LNet: Removed LNI 192.168.206.152@tcp49 [14849.398844] LNet: Removed LNI 192.168.206.152@tcp50 [14850.295213] Key type .llcrypt unregistered [14850.298452] Key type ._llcrypt unregistered [14872.288908] Key type ._llcrypt registered [14872.295166] Key type .llcrypt registered [14872.416906] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [14886.248352] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [14887.194528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14887.214535] alg: No test for adler32 (adler32-zlib) [14888.207411] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [14888.541462] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [14890.192317] Key type lgssc registered [14891.200450] Lustre: Echo OBD driver; http://www.lustre.org/ [14923.263512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14923.312349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14924.566169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14924.601617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14924.749046] Lustre: lustre-MDT0000: new disk, initializing [14924.869929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14924.891315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14928.908292] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14939.268555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14939.381244] Lustre: 505440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14939.432262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14939.435908] Lustre: Skipped 1 previous similar message [14939.518246] Lustre: lustre-MDT0001: new disk, initializing [14939.688580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14939.725893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14939.748038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14945.182541] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14959.221198] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14966.124988] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14975.546341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14975.769780] Lustre: *** cfs_fail_loc=90f, val=0*** [14975.867493] Lustre: lustre-OST0000: new disk, initializing [14975.870622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14975.876731] Lustre: 507458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14975.950835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14983.790935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14983.814976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14984.058077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14984.776436] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [14993.714272] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15004.648431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15004.663888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15004.665463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15006.846849] Lustre: server umount lustre-OST0000 complete [15011.297820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15011.304911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15011.322449] Lustre: Skipped 1 previous similar message [15011.326754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15011.332251] Lustre: Skipped 1 previous similar message [15014.881693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15014.882018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15016.910996] Lustre: server umount lustre-MDT0000 complete [15020.488833] LustreError: 505432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563452 with bad export cookie 13061583160161819111 [15020.494339] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15020.499371] LustreError: 505432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15020.892920] Lustre: server umount lustre-MDT0001 complete [15030.552912] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [15033.436615] Key type lgssc unregistered [15033.744375] LNet: 509368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15033.755073] LNetError: 509368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15033.774154] LNet: Removed LNI 192.168.206.152@tcp [15034.555291] Key type .llcrypt unregistered [15034.557159] Key type ._llcrypt unregistered [15053.064698] Key type ._llcrypt registered [15053.067296] Key type .llcrypt registered [15053.163237] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [15054.284754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15054.319747] alg: No test for adler32 (adler32-zlib) [15055.349755] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [15055.654306] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [15057.320212] Key type lgssc registered [15058.570946] Lustre: Echo OBD driver; http://www.lustre.org/ [15066.046908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15066.087851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15067.603605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15072.148543] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [15076.901116] LustreError: 510641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15076.913690] LustreError: 510641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15080.271493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15080.729884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15085.185448] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [15092.401980] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15097.607676] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15105.168953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15105.594221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15112.664575] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [15120.621225] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15130.688955] loop0: detected capacity change from 0 to 204800000 [15130.991757] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15131.023095] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15131.037435] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15137.244894] loop0: detected capacity change from 0 to 204800000 [15137.451034] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15137.468275] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15137.478821] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15142.219835] loop0: detected capacity change from 0 to 204800000 [15147.826163] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15148.055136] Lustre: fs15246-OST0000: new disk, initializing [15148.064432] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15148.078457] Lustre: 514232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15148.144851] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15154.393274] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [15164.016880] loop1: detected capacity change from 0 to 204800000 [15168.939029] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15169.066610] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15169.137423] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15169.245200] Lustre: fs15246-MDT0000: new disk, initializing [15169.425125] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15169.451708] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15169.603431] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15170.658843] Lustre: 515355:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000252e363d x1874396170991360/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787563618 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15174.034091] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [15179.747582] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15179.766098] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15179.795870] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15196.662924] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15196.681825] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15201.582795] Lustre: server umount fs15246-MDT0000 complete [15201.787717] LustreError: 510637:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15206.886921] LustreError: 511673:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15212.014189] LustreError: 510635:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15212.534465] Lustre: server umount fs15246-OST0000 complete [15227.363351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15227.388484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15227.392967] Lustre: Skipped 1 previous similar message [15232.484481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15232.505542] Lustre: Skipped 1 previous similar message [15233.609432] Lustre: server umount lustre-OST0000 complete [15239.649372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15239.658538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15239.673847] Lustre: Skipped 1 previous similar message [15239.689138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15242.733257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15244.907158] Lustre: server umount lustre-MDT0000 complete [15247.843574] LustreError: 510635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15247.881565] LustreError: 510635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15249.076047] LustreError: 510620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563681 with bad export cookie 2934746246915095988 [15249.076853] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15249.086584] LustreError: 510620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15249.359681] Lustre: server umount lustre-MDT0001 complete [15262.562730] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [15266.412695] Key type lgssc unregistered [15266.780694] LNet: 518458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15266.787928] LNetError: 518458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15266.810155] LNet: Removed LNI 192.168.206.152@tcp [15267.902147] Key type .llcrypt unregistered [15267.907460] Key type ._llcrypt unregistered [15287.622377] Key type ._llcrypt registered [15287.625934] Key type .llcrypt registered [15287.760540] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [15288.730806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15288.906911] alg: No test for adler32 (adler32-zlib) [15290.048150] Lustre: Lustre: Build Version: 2.17.57_82_g53f9976 [15290.318958] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [15292.048171] Key type lgssc registered [15293.096678] Lustre: Echo OBD driver; http://www.lustre.org/ [15331.953513] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [15341.173768] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [15372.548940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15372.576848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15373.925399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15373.950624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15374.068932] Lustre: lustre-MDT0000: new disk, initializing [15374.169846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15374.204467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15378.337143] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [15388.692769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15388.883227] Lustre: 522926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15388.958983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15388.965835] Lustre: Skipped 1 previous similar message [15389.098369] Lustre: lustre-MDT0001: new disk, initializing [15389.123727] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15389.208675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15389.245633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15389.258297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15393.744797] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [15403.113938] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15408.675479] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15415.569158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15415.791208] Lustre: lustre-OST0000: new disk, initializing [15415.799114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15415.807632] Lustre: 524895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15415.904904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15421.865837] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [15425.158212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15425.171882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15425.330694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15431.612674] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15436.421850] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15436.613325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15440.774928] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15441.046491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15445.993565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15445.998047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15446.006689] LustreError: Skipped 1 previous similar message [15446.013664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15446.024406] Lustre: Skipped 1 previous similar message [15446.046089] Lustre: Skipped 1 previous similar message [15451.119068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15451.129896] Lustre: Skipped 1 previous similar message [15456.228632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15456.239104] Lustre: Skipped 1 previous similar message [15457.248311] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15457.448106] Lustre: server umount lustre-OST0000 complete [15461.346194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15461.348776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15461.366911] Lustre: Skipped 1 previous similar message [15461.390126] Lustre: Skipped 1 previous similar message [15467.734136] Lustre: server umount lustre-MDT0000 complete [15471.787356] LustreError: 522919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563904 with bad export cookie 14142513562971158002 [15471.792310] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15471.805400] LustreError: 522919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15472.150882] Lustre: server umount lustre-MDT0001 complete [15497.907963] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [15507.284810] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [15541.935264] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15200 sec ========= 05:32:53 (1787563973) [15544.006990] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:32:55 (1787563975) === [15547.978325] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:32:59 (1787563979) === [15591.370666] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [15594.260963] Key type lgssc unregistered [15594.762486] LNet: 531542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15594.777220] LNetError: 531542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15594.806334] LNet: Removed LNI 192.168.206.152@tcp [15595.865882] Key type .llcrypt unregistered [15595.872617] Key type ._llcrypt unregistered