[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508655668 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002394] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010100] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.022392] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026840] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028093] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031126] *** VALIDATE tmpfs *** [ 0.033178] *** VALIDATE proc *** [ 0.034234] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042854] debug: unmapping init [mem 0xffffffff8bc59000-0xffffffff8bc60fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044648] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052279] rcu: Hierarchical SRCU implementation. [ 0.054384] smp: Bringing up secondary CPUs ... [ 0.055573] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.059484] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132899] node 0 deferred pages initialised in 68ms [ 0.136124] devtmpfs: initialized [ 0.137215] x86/mm: Memory block size: 128MB [ 0.139438] gcov: version magic: 0x41383552 [ 0.140688] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142136] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145320] pinctrl core: initialized pinctrl subsystem [ 0.147202] [ 0.147609] ************************************************************* [ 0.150010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152012] ** ** [ 0.155014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158012] ** ** [ 0.160010] ** This means that this kernel is built to expose internal ** [ 0.162011] ** IOMMU data structures, which may compromise security on ** [ 0.165025] ** your system. ** [ 0.168012] ** ** [ 0.170010] ** If you see this message and you are not debugging the ** [ 0.172013] ** kernel, report this immediately to your vendor! ** [ 0.175012] ** ** [ 0.177011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179011] ************************************************************* [ 0.181635] NET: Registered protocol family 16 [ 0.183411] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.185052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.192013] cpuidle: using governor menu [ 0.193901] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196315] PCI: Using configuration type 1 for base access [ 0.199140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.208051] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213219] cryptd: max_cpu_qlen set to 1000 [ 0.215246] ACPI: Added _OSI(Module Device) [ 0.216023] ACPI: Added _OSI(Processor Device) [ 0.217000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.231440] ACPI: Interpreter enabled [ 0.232054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.234014] ACPI: Using IOAPIC for interrupt routing [ 0.236121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239421] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.251311] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.256019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.265248] acpiphp: Slot [2] registered [ 0.266092] acpiphp: Slot [5] registered [ 0.268090] acpiphp: Slot [6] registered [ 0.270123] acpiphp: Slot [7] registered [ 0.271103] acpiphp: Slot [8] registered [ 0.272000] acpiphp: Slot [9] registered [ 0.272000] acpiphp: Slot [10] registered [ 0.274114] acpiphp: Slot [3] registered [ 0.276099] acpiphp: Slot [4] registered [ 0.277123] acpiphp: Slot [11] registered [ 0.279103] acpiphp: Slot [12] registered [ 0.281140] acpiphp: Slot [13] registered [ 0.282099] acpiphp: Slot [14] registered [ 0.284091] acpiphp: Slot [15] registered [ 0.286103] acpiphp: Slot [16] registered [ 0.287103] acpiphp: Slot [17] registered [ 0.289101] acpiphp: Slot [18] registered [ 0.291123] acpiphp: Slot [19] registered [ 0.292118] acpiphp: Slot [20] registered [ 0.293088] acpiphp: Slot [21] registered [ 0.295072] acpiphp: Slot [22] registered [ 0.296093] acpiphp: Slot [23] registered [ 0.297092] acpiphp: Slot [24] registered [ 0.299126] acpiphp: Slot [25] registered [ 0.300102] acpiphp: Slot [26] registered [ 0.302117] acpiphp: Slot [27] registered [ 0.303097] acpiphp: Slot [28] registered [ 0.305102] acpiphp: Slot [29] registered [ 0.306128] acpiphp: Slot [30] registered [ 0.308127] acpiphp: Slot [31] registered [ 0.309060] PCI host bridge to bus 0000:00 [ 0.311019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.314023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.322024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326963] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330082] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339739] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344924] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.347027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.349016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354545] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.357687] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.360045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362810] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.382000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392013] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.400016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.411014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.427016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439936] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.458024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.481017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489538] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.496018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.503019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.521022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.533012] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.548016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.570020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.579179] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.586019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.593021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.609022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.622697] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.633017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.642030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.668016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.684903] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.687422] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.689521] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.691410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.692000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.695055] iommu: Default domain type: Passthrough [ 0.696000] SCSI subsystem initialized [ 0.696207] ACPI: bus type USB registered [ 0.698132] usbcore: registered new interface driver usbfs [ 0.700137] usbcore: registered new interface driver hub [ 0.702120] usbcore: registered new device driver usb [ 0.703194] pps_core: LinuxPPS API ver. 1 registered [ 0.705019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.706080] PTP clock support registered [ 0.708102] EDAC MC: Ver: 3.0.0 [ 0.711166] PCI: Using ACPI for IRQ routing [ 0.712606] NetLabel: Initializing [ 0.716021] NetLabel: domain hash size = 128 [ 0.717012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.719113] NetLabel: unlabeled traffic allowed by default [ 0.720123] vgaarb: loaded [ 0.722306] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.723000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.728000] clocksource: Switched to clocksource kvm-clock [ 0.841588] VFS: Disk quotas dquot_6.6.0 [ 0.843291] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.845717] *** VALIDATE ramfs *** [ 0.847086] *** VALIDATE hugetlbfs *** [ 0.848821] pnp: PnP ACPI init [ 0.851583] pnp: PnP ACPI: found 6 devices [ 0.890249] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.894075] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.896241] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.898781] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.900992] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.903266] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.906634] NET: Registered protocol family 2 [ 0.909291] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.914344] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.918315] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.924091] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.927787] TCP: Hash tables configured (established 65536 bind 65536) [ 0.930996] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.934600] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.937873] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941016] NET: Registered protocol family 1 [ 0.943968] RPC: Registered named UNIX socket transport module. [ 0.946021] RPC: Registered udp transport module. [ 0.947462] RPC: Registered tcp transport module. [ 0.948730] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.951247] NET: Registered protocol family 44 [ 0.953104] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.955513] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.958376] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.960838] PCI: CLS 0 bytes, default 64 [ 0.962632] Unpacking initramfs... [ 2.406820] debug: unmapping init [mem 0xffff9f153cc54000-0xffff9f153ffbffff] [ 2.412797] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.415337] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.419627] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.937188] Initialise system trusted keyrings [ 2.938371] Key type blacklist registered [ 2.939734] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.951431] zbud: loaded [ 2.955070] *** VALIDATE nfs *** [ 2.956226] *** VALIDATE nfs4 *** [ 2.957868] pstore: using deflate compression [ 2.961618] Platform Keyring initialized [ 3.069729] NET: Registered protocol family 38 [ 3.070997] Key type asymmetric registered [ 3.072363] Asymmetric key parser 'x509' registered [ 3.073519] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.077098] io scheduler mq-deadline registered [ 3.078698] io scheduler kyber registered [ 3.081466] io scheduler bfq registered [ 3.084608] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.088990] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.092284] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.095583] ACPI: Power Button [PWRF] [ 3.101555] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.108889] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.129200] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.146172] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.169953] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.198083] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.227235] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.232980] Non-volatile memory driver v1.3 [ 3.234442] Linux agpgart interface v0.103 [ 3.268644] virtio_blk virtio1: [vda] 149376 512-byte logical blocks (76.5 MB/72.9 MiB) [ 3.271393] vda: detected capacity change from 0 to 76480512 [ 3.285929] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.288760] vdb: detected capacity change from 0 to 1073741824 [ 3.303910] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.306956] vdc: detected capacity change from 0 to 2621440000 [ 3.323378] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.326842] vdd: detected capacity change from 0 to 2621440000 [ 3.345167] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.348194] vde: detected capacity change from 0 to 4294967296 [ 3.360849] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.363711] vdf: detected capacity change from 0 to 4294967296 [ 3.369862] libphy: Fixed MDIO Bus: probed [ 3.375102] usbcore: registered new interface driver usbserial_generic [ 3.377865] usbserial: USB Serial support registered for generic [ 3.380668] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.386288] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.388385] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.391376] mousedev: PS/2 mouse device common for all mice [ 3.394240] rtc_cmos 00:05: RTC can wake from S4 [ 3.396761] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.397601] rtc_cmos 00:05: registered as rtc0 [ 3.402422] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.405071] intel_pstate: CPU model not supported [ 3.408165] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.411060] hid: raw HID events driver (C) Jiri Kosina [ 3.415357] usbcore: registered new interface driver usbhid [ 3.417794] usbhid: USB HID core driver [ 3.418057] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.419638] drop_monitor: Initializing network drop monitor service [ 3.424993] Initializing XFRM netlink socket [ 3.427286] NET: Registered protocol family 10 [ 3.430493] Segment Routing with IPv6 [ 3.431864] NET: Registered protocol family 17 [ 3.434250] mpls_gso: MPLS GSO support [ 3.440718] RAS: Correctable Errors collector initialized. [ 3.443165] AVX version of gcm_enc/dec engaged. [ 3.444645] AES CTR mode by8 optimization enabled [ 3.540594] sched_clock: Marking stable (3540572475, 0)->(4449270530, -908698055) [ 3.544492] registered taskstats version 1 [ 3.546918] Loading compiled-in X.509 certificates [ 3.550650] zswap: loaded using pool lzo/zbud [ 3.580566] Key type big_key registered [ 3.592470] Key type encrypted registered [ 3.594139] ima: No TPM chip found, activating TPM-bypass! [ 3.596347] ima: Allocated hash algorithm: sha1 [ 3.598087] ima: No architecture policies found [ 3.599868] evm: Initialising EVM extended attributes: [ 3.601600] evm: security.selinux [ 3.602533] evm: security.ima [ 3.603566] evm: security.capability [ 3.604884] evm: HMAC attrs: 0x1 [ 3.607146] rtc_cmos 00:05: setting system clock to 2026-08-22 10:15:10 UTC (1787393710) [ 3.614553] debug: unmapping init [mem 0xffffffff8cc03000-0xffffffff8cdfffff] [ 3.618215] debug: unmapping init [mem 0xffffffff8b982000-0xffffffff8bc58fff] [ 3.627087] Write protecting the kernel read-only data: 28672k [ 3.631174] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 3.634058] debug: unmapping init [mem 0xffffffff8a914000-0xffffffff8a9fffff] [ 3.674830] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.683634] systemd[1]: Detected virtualization kvm. [ 3.685873] systemd[1]: Detected architecture x86-64. [ 3.687885] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.720096] systemd[1]: No hostname configured. [ 3.721862] systemd[1]: Set hostname to . [ 3.723953] random: systemd: uninitialized urandom read (16 bytes read) [ 3.726929] systemd[1]: Initializing machine ID from random generator. [ 3.875615] random: systemd: uninitialized urandom read (16 bytes read) [ 3.878863] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.885069] random: systemd: uninitialized urandom read (16 bytes read) [ 3.888174] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.893906] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.567922] device-mapper: uevent: version 1.0.3 [ 4.570058] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.359142] virtio_net virtio0 ens2: renamed from eth0 [ 5.377063] random: fast init done [ 5.388055] scsi host0: ata_piix [ 5.458913] scsi host1: ata_piix [ 5.460258] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.462297] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.123520] dracut-initqueue[577]: RTNETLINK answers: File exists [ 10.112436] random: crng init done [ 10.113832] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.608755] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.778889] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.046363] SELinux: Disabled at runtime. [ 12.113848] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.121291] systemd[1]: Detected virtualization kvm. [ 12.122900] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.666523] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.669992] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.677211] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.681212] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.684688] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.693246] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.697321] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.787400] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.219179] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.487838] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.596731] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.675920] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.688973] EDAC sbridge: Ver: 1.1.2 [ 15.682247] Key type dns_resolver registered [ 15.991355] NFS: Registering the id_resolver key type [ 15.993368] Key type id_resolver registered [ 15.995013] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg655-server login: [ 45.993797] hrtimer: interrupt took 12786755 ns [ 69.598658] libcfs: loading out-of-tree module taints kernel. [ 69.678550] Key type ._llcrypt registered [ 69.687183] Key type .llcrypt registered [ 69.830503] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 90.689837] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 92.422750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.458723] alg: No test for adler32 (adler32-zlib) [ 93.900843] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 94.990572] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 96.727181] Key type lgssc registered [ 98.510093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.388727] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.914793] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 168.314558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.369872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.660665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.706414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.829293] Lustre: lustre-MDT0000: new disk, initializing [ 169.939833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.962831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.475974] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.233829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.298418] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.324154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.327884] Lustre: Skipped 1 previous similar message [ 187.387358] Lustre: lustre-MDT0001: new disk, initializing [ 187.433098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.454203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.458074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.921664] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.521055] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 207.048509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.419704] Lustre: lustre-OST0000: new disk, initializing [ 207.426910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.434159] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.520538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.942630] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.120773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 216.130805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 216.208839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 228.412928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.534871] Lustre: lustre-OST0001: new disk, initializing [ 228.537656] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 228.541269] Lustre: 9518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.595548] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.013385] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.124127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 237.140608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 237.204023] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 247.753936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.876974] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 265.801941] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.755351] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing yml_node [ 274.777253] Lustre: DEBUG MARKER: Client: 2.17.54.107 [ 277.382846] Lustre: DEBUG MARKER: MDS: 2.17.54.107 [ 279.971532] Lustre: DEBUG MARKER: OSS: 2.17.54.107 [ 281.708537] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 06:19:45 EDT 2026 [ 299.316579] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 300.891981] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 319.462050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.462424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.486382] Lustre: Skipped 1 previous similar message [ 323.453263] Lustre: server umount lustre-MDT0000 complete [ 327.240857] LustreError: 7466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394034 with bad export cookie 10830635229817979186 [ 327.242097] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.250445] LustreError: 7466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 329.696028] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.696915] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.703314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.703323] Lustre: Skipped 3 previous similar messages [ 329.716523] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 329.751056] Lustre: Skipped 3 previous similar messages [ 332.706324] LustreError: 12158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.712118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.723430] LustreError: 12158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 332.744048] Lustre: Skipped 2 previous similar messages [ 333.628463] Lustre: server umount lustre-MDT0001 complete [ 337.962104] Lustre: server umount lustre-OST0000 complete [ 341.402864] Lustre: server umount lustre-OST0001 complete [ 347.558828] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 356.856777] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 387.766417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.954301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 388.010686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 388.124376] Lustre: lustre-MDT0000: new disk, initializing [ 388.262947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.295732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 392.336883] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 403.002713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.205422] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 403.379450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 403.387455] Lustre: Skipped 1 previous similar message [ 403.540918] Lustre: lustre-MDT0001: new disk, initializing [ 403.711229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 403.781632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 403.791188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.127583] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 417.839752] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 422.750530] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 429.556405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.799083] Lustre: lustre-OST0000: new disk, initializing [ 429.803399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.807842] Lustre: 17042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.875815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.814114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.822635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.922441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 436.155692] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 445.186289] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 449.296714] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.492643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 455.034569] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 455.293915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.702790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.704353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.707147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.716874] LustreError: Skipped 1 previous similar message [ 457.733117] Lustre: Skipped 1 previous similar message [ 462.826395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.833661] Lustre: Skipped 2 previous similar messages [ 463.789892] Lustre: server umount lustre-OST0000 complete [ 467.936741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.961200] Lustre: Skipped 1 previous similar message [ 473.057968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 473.069583] Lustre: Skipped 3 previous similar messages [ 482.271380] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 482.620606] Lustre: server umount lustre-MDT0000 complete [ 483.306590] LustreError: 16424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.332416] LustreError: 16424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 486.018331] LustreError: 15868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394192 with bad export cookie 10830635229817980719 [ 486.020560] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.032535] LustreError: 15868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 486.288290] Lustre: server umount lustre-MDT0001 complete [ 489.813248] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:23:14 (1787394194) [ 491.795091] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 493.237743] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:23:17 (1787394197) [ 500.165305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.626871] LustreError: 19450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.718632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.291227] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 505.827442] LustreError: 19451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.244857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.871355] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 526.346741] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 531.912643] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 538.324785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.709620] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 552.537621] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 560.609708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 560.620804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.631459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 560.641506] Lustre: Skipped 2 previous similar messages [ 561.634063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 566.192754] Lustre: server umount lustre-OST0000 complete [ 570.848322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 570.857954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.872853] Lustre: Skipped 1 previous similar message [ 576.305625] Lustre: server umount lustre-MDT0000 complete [ 577.009867] LustreError: 19450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.028741] LustreError: 19450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 580.024287] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394286 with bad export cookie 10830635229817981601 [ 580.029936] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 580.038282] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 580.544204] Lustre: server umount lustre-MDT0001 complete [ 590.079508] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 592.924462] Key type lgssc unregistered [ 593.257128] LNet: 22984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.271257] LNetError: 22984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.298967] LNet: Removed LNI 192.168.206.155@tcp [ 594.752251] Key type .llcrypt unregistered [ 594.756862] Key type ._llcrypt unregistered [ 618.364919] Key type ._llcrypt registered [ 618.367630] Key type .llcrypt registered [ 618.470450] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 619.173703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 619.317309] alg: No test for adler32 (adler32-zlib) [ 620.410802] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 620.683372] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 622.319164] Key type lgssc registered [ 623.419949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.012071] loop: module loaded [ 631.095167] loop0: detected capacity change from 0 to 204800000 [ 631.342547] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.351860] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.717960] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.975191] loop0: detected capacity change from 0 to 204800000 [ 637.158889] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.170802] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.178511] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.030261] loop0: detected capacity change from 0 to 204800000 [ 648.340333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 648.390592] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.644682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 649.689617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 649.771502] Lustre: lustre-MDT0000: new disk, initializing [ 649.842996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.864289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 654.210988] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 662.657268] loop1: detected capacity change from 0 to 204800000 [ 668.731480] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.110047] Lustre: lustre-OST07c6: new disk, initializing [ 669.116644] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 669.126815] Lustre: Skipped 1 previous similar message [ 669.134508] Lustre: 26171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 669.253490] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 676.959532] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 678.485759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 678.504485] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 678.566976] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 682.180477] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 682.195287] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 682.207382] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 682.215687] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 682.224328] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 682.233575] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 698.327386] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 698.341774] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.349116] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.356677] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 698.363090] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 698.368065] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.168328] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 703.182811] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 703.186939] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 703.191842] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.196510] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.201780] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.207076] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 703.213497] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.220652] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 703.226231] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.230862] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.236179] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.368797] Lustre: 26179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 706.375743] Lustre: 26179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 706.391129] Lustre: 26179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 706.397688] Lustre: 26179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.409928] Lustre: 26179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 706.418367] Lustre: 26179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.424719] Lustre: 26179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 706.432190] Lustre: 26179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.441119] Lustre: 26179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 706.445803] Lustre: 26179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.458342] Lustre: 26179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 706.466482] Lustre: 26179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.028864] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 711.041888] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 711.053514] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 711.063378] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.075127] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 711.085231] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.094724] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 711.104493] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.114348] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 711.123846] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.134071] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 711.144414] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.829909] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 720.835963] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 720.840852] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 720.845597] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.850310] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.854956] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.858855] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 720.863052] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.866944] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 720.871668] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.877902] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.885523] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 770.534666] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 770.539296] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.587383] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 775.652055] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 780.768743] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 784.351328] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 784.547383] Lustre: server umount lustre-OST07c6 complete [ 793.692889] Lustre: server umount lustre-MDT0000 complete [ 810.748167] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 813.609678] Key type lgssc unregistered [ 813.971488] LNet: 28750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.979332] LNetError: 28750:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.014966] LNet: Removed LNI 192.168.206.155@tcp [ 815.955939] Key type .llcrypt unregistered [ 815.961047] Key type ._llcrypt unregistered [ 834.775179] Key type ._llcrypt registered [ 834.777319] Key type .llcrypt registered [ 834.940437] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 836.608435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 836.618336] alg: No test for adler32 (adler32-zlib) [ 837.602925] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 837.895913] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 839.655261] Key type lgssc registered [ 840.687455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.206912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 850.241226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.710489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.190704] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 861.161142] LustreError: 30027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 861.183105] LustreError: 30027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 866.285568] LustreError: 30026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.390041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.861918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 874.804899] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 883.195427] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 889.335737] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 896.771432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.235372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.342820] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 910.245877] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 921.772350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 935.395276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 935.405753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.420050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 937.442480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 937.460632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 937.469491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.970512] Lustre: server umount lustre-OST0000 complete [ 946.656458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 946.668117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.692913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.707620] Lustre: Skipped 1 previous similar message [ 950.880640] Lustre: server umount lustre-MDT0000 complete [ 952.801077] LustreError: 31226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.824586] LustreError: 31226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 954.191302] LustreError: 30007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394661 with bad export cookie 9102097216104509692 [ 954.217489] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.465685] Lustre: server umount lustre-MDT0001 complete [ 965.092491] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 967.877370] Key type lgssc unregistered [ 968.087751] LNet: 33787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.096817] LNetError: 33787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.123397] LNet: Removed LNI 192.168.206.155@tcp [ 969.006916] Key type .llcrypt unregistered [ 969.008261] Key type ._llcrypt unregistered [ 986.975571] Key type ._llcrypt registered [ 986.978947] Key type .llcrypt registered [ 987.085951] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 988.010896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.055691] alg: No test for adler32 (adler32-zlib) [ 989.107308] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 989.341797] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 991.039193] Key type lgssc registered [ 992.071343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.640151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.661941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.222592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.819836] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1011.689601] LustreError: 35054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.709559] LustreError: 35054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1013.123541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.686639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1017.653681] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1024.541848] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1030.535662] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1036.752525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1037.126288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.656437] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1046.356640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1048.065360] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1058.789353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1058.796100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.819723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1061.867343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.870553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1061.870792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.746358] Lustre: server umount lustre-OST0000 complete [ 1070.049907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1070.059885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.070273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1070.078192] Lustre: Skipped 1 previous similar message [ 1072.096397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.098817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.120422] Lustre: Skipped 1 previous similar message [ 1073.351183] Lustre: server umount lustre-MDT0000 complete [ 1076.234826] LustreError: 35034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394783 with bad export cookie 1215415229116390519 [ 1076.236968] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1076.247401] LustreError: 35034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1076.459831] Lustre: server umount lustre-MDT0001 complete [ 1085.200401] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1087.871627] Key type lgssc unregistered [ 1088.137844] LNet: 38590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.148260] LNetError: 38590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.171775] LNet: Removed LNI 192.168.206.155@tcp [ 1089.253110] Key type .llcrypt unregistered [ 1089.255636] Key type ._llcrypt unregistered [ 1110.298605] Key type ._llcrypt registered [ 1110.300814] Key type .llcrypt registered [ 1110.395084] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1111.182327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.265257] alg: No test for adler32 (adler32-zlib) [ 1112.376341] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1112.759675] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1114.455760] Key type lgssc registered [ 1115.767361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1125.075414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1125.096375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.727179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.479044] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1137.127888] LustreError: 39986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.142633] LustreError: 39986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1141.814136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.401918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1147.054629] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1153.190879] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1158.563121] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1166.053929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1166.497577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.664656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1172.936635] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1180.466547] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1188.151312] Lustre: server umount lustre-MDT0000 complete [ 1188.328309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1188.335373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.345812] LustreError: 39986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.361635] Lustre: Skipped 1 previous similar message [ 1191.963717] LustreError: 39968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394898 with bad export cookie 9965138246312955269 [ 1191.967932] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1191.971723] LustreError: 39968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1193.441200] LustreError: 39982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.446797] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.459805] LustreError: 39982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1193.470122] Lustre: Skipped 1 previous similar message [ 1193.473601] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1196.451437] LustreError: 39983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.451743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1196.479894] LustreError: 39983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1196.490037] Lustre: Skipped 1 previous similar message [ 1198.456884] Lustre: server umount lustre-MDT0001 complete [ 1219.627163] Lustre: server umount lustre-OST0000 complete [ 1234.885319] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1237.791920] Key type lgssc unregistered [ 1238.055764] LNet: 43844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.066385] LNetError: 43844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.085788] LNet: Removed LNI 192.168.206.155@tcp [ 1239.059249] Key type .llcrypt unregistered [ 1239.061301] Key type ._llcrypt unregistered [ 1262.034965] Key type ._llcrypt registered [ 1262.037870] Key type .llcrypt registered [ 1262.172516] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1263.279443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.414101] alg: No test for adler32 (adler32-zlib) [ 1264.516382] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1264.739529] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1266.439162] Key type lgssc registered [ 1267.783362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1275.673844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.685823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.143210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.337990] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1287.668612] LustreError: 45261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.707427] LustreError: 45261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1290.334605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.822972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1295.385870] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1302.329535] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1308.152983] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1314.412968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1314.645427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1320.315235] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1322.876527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1326.804349] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1334.405816] Lustre: lustre-OST0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 1336.288653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.299615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.314230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1338.346708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.348704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1338.366501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1338.369231] Lustre: Skipped 1 previous similar message [ 1339.521425] LustreError: 47081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.756819] Lustre: server umount lustre-OST0000 complete [ 1357.286145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1357.296304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.309941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1363.622805] Lustre: server umount lustre-MDT0000 complete [ 1363.941312] LustreError: 45261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.966735] LustreError: 45261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1367.125149] LustreError: 45243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395074 with bad export cookie 8353836387380253363 [ 1367.129877] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.138533] LustreError: 45243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1373.461810] Lustre: server umount lustre-MDT0001 complete [ 1383.384429] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1385.671456] Key type lgssc unregistered [ 1385.986546] LNet: 48968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.993051] LNetError: 48968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.015995] LNet: Removed LNI 192.168.206.155@tcp [ 1386.799413] Key type .llcrypt unregistered [ 1386.801484] Key type ._llcrypt unregistered [ 1408.240674] Key type ._llcrypt registered [ 1408.242875] Key type .llcrypt registered [ 1408.364962] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1409.465584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.611018] alg: No test for adler32 (adler32-zlib) [ 1410.714250] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1410.984334] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1412.671151] Key type lgssc registered [ 1413.860068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1421.181369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1421.198280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.644982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.041514] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1433.067589] LustreError: 50386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.089679] LustreError: 50386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1435.314589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.730236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1440.467860] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1448.216935] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1454.516055] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1461.557775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1461.871041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.015429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1468.489782] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1476.588588] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1485.073923] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1488.803205] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1501.481387] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 6h40m04s [ 1503.267123] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1506.273767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.280884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1506.294187] Lustre: Skipped 2 previous similar messages [ 1506.318779] Lustre: Skipped 2 previous similar messages [ 1507.363529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.003142] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 1509.013061] Lustre: Skipped 2 previous similar messages [ 1511.639332] Lustre: server umount lustre-MDT0000 complete [ 1513.451034] LustreError: 50382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1513.482976] LustreError: 50382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1515.336046] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1516.140764] LustreError: 50380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1519.594424] LustreError: 51113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1519.623836] LustreError: 51113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1522.823207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.039918] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.402212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.493050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1524.492144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1528.903547] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1532.736809] Lustre: DEBUG MARKER: Wait for df (36567) ... [ 1534.505492] Lustre: DEBUG MARKER: done [ 1540.069685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.085528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1540.090114] Lustre: Skipped 1 previous similar message [ 1545.192247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1545.205513] Lustre: Skipped 1 previous similar message [ 1545.785347] Lustre: server umount lustre-OST0000 complete [ 1550.310853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.311935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1550.327372] Lustre: Skipped 1 previous similar message [ 1555.771564] Lustre: server umount lustre-MDT0000 complete [ 1556.450693] LustreError: 52790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.462635] LustreError: 52790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1559.808629] LustreError: 52229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395266 with bad export cookie 12525424446160331620 [ 1559.818779] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1559.819907] LustreError: 52229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1560.300884] Lustre: server umount lustre-MDT0001 complete [ 1571.372873] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1574.279533] Key type lgssc unregistered [ 1574.632814] LNet: 55307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.646171] LNetError: 55307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.663573] LNet: Removed LNI 192.168.206.155@tcp [ 1575.776512] Key type .llcrypt unregistered [ 1575.778637] Key type ._llcrypt unregistered [ 1618.563754] Key type ._llcrypt registered [ 1618.571805] Key type .llcrypt registered [ 1618.761447] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1619.905635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1620.085763] alg: No test for adler32 (adler32-zlib) [ 1621.184944] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1621.415258] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1623.151187] Key type lgssc registered [ 1624.248387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1632.076871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1632.097135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.397908] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1633.413297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1633.521775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1638.015918] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1646.781936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.883357] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1646.911146] Lustre: 57516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1647.174883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1652.114541] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1659.732680] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1665.683732] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1672.892318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1673.056784] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1673.256924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.545475] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1682.644550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1688.521488] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1695.705586] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 6h43m19s [ 1697.608855] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1704.688787] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1707.205814] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1707.269390] LustreError: 57522:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1727.651137] Lustre: lustre-MDT0000: Client 96f497d7-1b1c-452a-aad3-983f5a12a07a (at 192.168.206.55@tcp) reconnecting [ 1747.295112] LustreError: 57522:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1749.621743] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1752.184990] Lustre: DEBUG MARKER: done [ 1759.730777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1759.732822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1759.744202] LustreError: Skipped 1 previous similar message [ 1759.747878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1759.756495] Lustre: Skipped 1 previous similar message [ 1759.777376] Lustre: Skipped 1 previous similar message [ 1763.553827] Lustre: server umount lustre-OST0000 complete [ 1769.957518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.958243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1769.961567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1769.984320] Lustre: Skipped 1 previous similar message [ 1771.106898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1771.116495] Lustre: Skipped 3 previous similar messages [ 1773.619779] Lustre: server umount lustre-MDT0000 complete [ 1777.597992] LustreError: 59376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395484 with bad export cookie 8867456614771530931 [ 1777.599310] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1777.605228] LustreError: 59376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1778.141618] Lustre: server umount lustre-MDT0001 complete [ 1788.330438] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1790.640215] Key type lgssc unregistered [ 1790.921500] LNet: 61847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.933815] LNetError: 61847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.965813] LNet: Removed LNI 192.168.206.155@tcp [ 1791.889184] Key type .llcrypt unregistered [ 1791.893326] Key type ._llcrypt unregistered [ 1834.903248] Key type ._llcrypt registered [ 1834.906048] Key type .llcrypt registered [ 1834.973972] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1835.834873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.863730] alg: No test for adler32 (adler32-zlib) [ 1837.008779] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1837.322391] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1838.951333] Key type lgssc registered [ 1840.005300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1845.054038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1845.095747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.397816] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1846.422109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1846.578965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.422800] Lustre: Failing over lustre-MDT0000 [ 1847.609205] Lustre: server umount lustre-MDT0000 complete [ 1853.826696] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:45:58 (1787395558) [ 1861.116586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.728208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.310239] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1873.935873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.093541] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1874.139206] Lustre: 64642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1874.517608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.324315] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1883.587576] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1888.066480] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1894.193547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.419571] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1894.712330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.924071] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1903.263739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1907.973883] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1913.925966] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1919.902995] Lustre: server umount lustre-MDT0000 complete [ 1920.497196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1920.511482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.881478] LustreError: 64636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395630 with bad export cookie 10663316033635790413 [ 1923.896100] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1924.073820] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.088217] Lustre: Skipped 1 previous similar message [ 1924.107511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1924.662989] Lustre: server umount lustre-MDT0001 complete [ 1928.060198] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1932.492574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1941.585179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.576289] LustreError: 63565:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b04e3f90 x1874220069983488/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1949.043603] LustreError: 66490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1949.074661] LustreError: 66490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1949.147706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.272354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1954.051328] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1954.272911] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.408747] LustreError: 66490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.473254] LustreError: 68412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.484571] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1962.491341] LustreError: 68412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1962.971382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.401312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.174503] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1968.615765] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1975.232157] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1980.597191] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1989.089445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1989.095658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.098733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1989.117525] Lustre: Skipped 2 previous similar messages [ 1993.688629] Lustre: server umount lustre-MDT0000 complete [ 1994.214688] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.258725] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1997.444970] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395704 with bad export cookie 10663316033635792695 [ 1997.453555] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1997.463390] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1997.788151] Lustre: server umount lustre-MDT0001 complete [ 2004.005696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2013.556370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.714660] Lustre: 63568:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395706/real 1787395706] req@0000000028d6b4e6 x1874220070008576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787395722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2015.781094] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.847148] Lustre: 63568:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395709/real 1787395709] req@00000000f3b6ad79 x1874220070008704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787395725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2020.832481] Lustre: 63566:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395711/real 1787395711] req@00000000fefa8554 x1874220070008832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787395727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2022.944156] LustreError: 63565:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bbca3029 x1874220070009600/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2023.480423] LustreError: 66490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.592493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.680421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2024.543102] Lustre: 63567:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395714/real 1787395714] req@00000000c81935a0 x1874220070009088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787395730 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2028.304817] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2028.704419] Lustre: 63569:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395719/real 1787395719] req@0000000078fe2136 x1874220070009216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787395735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2036.824822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.657962] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2042.364533] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2044.461643] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2049.259713] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2055.149978] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2062.119737] Lustre: server umount lustre-MDT0000 complete [ 2062.819252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2062.827466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.841339] LustreError: 66490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.863576] LustreError: 66490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2065.884269] LustreError: 64634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395772 with bad export cookie 10663316033635794074 [ 2065.892381] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2065.896424] LustreError: 64634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2066.328573] Lustre: server umount lustre-MDT0001 complete [ 2073.192416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2077.203732] Lustre: DEBUG MARKER: files compared the same [ 2089.603458] Lustre: server umount lustre-OST0000 complete [ 2106.799757] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2110.059366] Key type lgssc unregistered [ 2110.318740] LNet: 74370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.321810] LNetError: 74370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.343623] LNet: Removed LNI 192.168.206.155@tcp [ 2111.285785] Key type .llcrypt unregistered [ 2111.287847] Key type ._llcrypt unregistered [ 2137.440518] Key type ._llcrypt registered [ 2137.442901] Key type .llcrypt registered [ 2137.537984] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2138.462323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2138.541443] alg: No test for adler32 (adler32-zlib) [ 2139.537765] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2139.764913] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2141.431945] Key type lgssc registered [ 2142.667487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.085088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2150.108906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.570259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.813450] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug +malloc all [ 2161.121232] LustreError: 75786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.139131] LustreError: 75786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2163.735296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.976698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2168.087558] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug +malloc all [ 2175.029318] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2180.312411] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2185.969579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2186.290332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.796467] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug +malloc all [ 2196.528876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2200.668618] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2211.809446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2211.819804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.826211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2211.840659] Lustre: Skipped 1 previous similar message [ 2211.877882] Lustre: Skipped 1 previous similar message [ 2216.520049] Lustre: server umount lustre-OST0000 complete [ 2222.055570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.083337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.089709] Lustre: Skipped 1 previous similar message [ 2227.170332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.173031] Lustre: Skipped 1 previous similar message [ 2231.266668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.274471] Lustre: Skipped 1 previous similar message [ 2235.359131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2235.520909] Lustre: server umount lustre-MDT0000 complete [ 2236.391940] LustreError: 77625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.417818] LustreError: 77625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2239.809936] LustreError: 75768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395946 with bad export cookie 1115882694597891872 [ 2239.821185] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.684371] Lustre: server umount lustre-MDT0001 complete [ 2251.875181] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2255.429321] Key type lgssc unregistered [ 2255.853688] LNet: 79318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.878414] LNetError: 79318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.937798] LNet: Removed LNI 192.168.206.155@tcp [ 2258.005928] Key type .llcrypt unregistered [ 2258.012219] Key type ._llcrypt unregistered [ 2282.049440] Key type ._llcrypt registered [ 2282.053977] Key type .llcrypt registered [ 2282.132676] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2282.920289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2282.979371] alg: No test for adler32 (adler32-zlib) [ 2284.041333] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2284.384258] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2286.064093] Key type lgssc registered [ 2287.599201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2296.648637] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2296.662548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2312.223372] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2317.345286] LustreError: 80708:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 2344.031339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.720734] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2356.883298] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2364.132626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.286102] Lustre: *** cfs_fail_loc=706, val=0*** [ 2364.663368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.732281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2369.525104] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2370.028868] LustreError: 80734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.050574] LustreError: 80734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2375.141326] LustreError: 80733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.219774] LustreError: 80734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.247112] LustreError: 80734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2378.417629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.787480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2383.785973] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2391.344690] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2396.896734] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2400.738067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2400.750272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.770134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2404.328154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2404.330808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2404.337864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.352400] Lustre: Skipped 1 previous similar message [ 2405.638864] Lustre: server umount lustre-OST0000 complete [ 2414.567269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2414.572092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.603156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.689156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.696312] Lustre: Skipped 2 previous similar messages [ 2424.287144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2424.498713] Lustre: server umount lustre-MDT0000 complete [ 2424.801712] LustreError: 82787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.817839] LustreError: 82787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2428.140726] LustreError: 82538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396135 with bad export cookie 8790660351552449953 [ 2428.144814] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2428.150844] LustreError: 82538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2428.513480] Lustre: server umount lustre-MDT0001 complete [ 2439.493537] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2442.598848] Key type lgssc unregistered [ 2442.830808] LNet: 84328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.835713] LNetError: 84328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.846832] LNet: Removed LNI 192.168.206.155@tcp [ 2443.667466] Key type .llcrypt unregistered [ 2443.670103] Key type ._llcrypt unregistered [ 2468.190449] Key type ._llcrypt registered [ 2468.192810] Key type .llcrypt registered [ 2468.337509] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2469.433426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2469.475818] alg: No test for adler32 (adler32-zlib) [ 2470.484133] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2470.723763] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2472.407154] Key type lgssc registered [ 2473.432981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2480.629308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2480.643928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.725912] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2492.986429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.625297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2498.273779] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2499.042078] LustreError: 86429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.063049] LustreError: 86429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2503.140200] LustreError: 86432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.164793] LustreError: 86432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2506.618808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2506.851117] LustreError: 86429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.869197] LustreError: 86429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2506.889714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.352164] LustreError: 86433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.375923] LustreError: 86433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2512.626580] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2517.475795] LustreError: 86432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.498044] LustreError: 86432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2520.254313] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2520.272180] Lustre: MGS: Client ff604e01-28a0-413e-8400-b12159f61c15 (at 0@lo) reconnecting [ 2520.294590] LustreError: 86443:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2520.535543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.639667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2525.265155] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2541.029318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2541.031345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.042139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2541.057034] Lustre: Skipped 1 previous similar message [ 2541.065824] Lustre: Skipped 1 previous similar message [ 2545.791530] Lustre: server umount lustre-OST0000 complete [ 2551.273601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.282230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.283628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2551.298519] Lustre: Skipped 1 previous similar message [ 2551.326187] Lustre: Skipped 1 previous similar message [ 2555.680584] Lustre: server umount lustre-MDT0000 complete [ 2556.387032] LustreError: 87984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.411820] LustreError: 87984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2560.542427] Lustre: server umount lustre-MDT0001 complete [ 2564.402449] Lustre: server umount MGS complete [ 2575.863929] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:57:59 (1787396279) [ 2605.807940] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 2615.244807] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2650.066098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.456772] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2663.326933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.526466] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2663.565357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2663.661839] Lustre: lustre-MDT0001: new disk, initializing [ 2663.738570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2663.766432] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2663.775073] Lustre: 93616:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2663.794636] LustreError: 93616:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2668.468265] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2678.446613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.727685] Lustre: lustre-OST0000: new disk, initializing [ 2678.731903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2678.738469] Lustre: 94382:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2678.793891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.047553] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2687.073976] Lustre: 94767:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b8155b92 x1874220733734528/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787396409 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2694.007887] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2700.650091] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.660923] Lustre: MGS: Client 42c6a72c-5c7f-4953-96b2-981787feacb3 (at 0@lo) reconnecting [ 2700.676188] LustreError: 93602:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2700.722236] Lustre: 92885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2700.770277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2700.866636] Lustre: lustre-MDT0000: new disk, initializing [ 2701.047410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.081970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2701.220614] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2702.178725] Lustre: 95451:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a909bbfe x1874220733741568/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787396425 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2705.373127] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2707.424277] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2711.280945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2711.300539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2711.387258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2737.131627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.148993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.177673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2739.171299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.179974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2739.187228] Lustre: Skipped 1 previous similar message [ 2741.758575] Lustre: server umount lustre-OST0000 complete [ 2749.411184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2749.411919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.421673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.455632] Lustre: Skipped 1 previous similar message [ 2751.513736] Lustre: server umount lustre-MDT0000 complete [ 2754.547987] LustreError: 93591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.614031] LustreError: 93591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2762.608633] Lustre: server umount lustre-MDT0001 complete [ 2767.026927] Lustre: server umount MGS complete [ 2777.636077] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:01:21 (1787396481) [ 2795.225116] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2797.338451] Key type lgssc unregistered [ 2797.731606] LNet: 97960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.738638] LNetError: 97960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2797.752555] LNet: Removed LNI 192.168.206.155@tcp [ 2798.732402] Key type .llcrypt unregistered [ 2798.734802] Key type ._llcrypt unregistered [ 2816.958460] Key type ._llcrypt registered [ 2816.973354] Key type .llcrypt registered [ 2817.152116] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2818.377940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2818.464401] alg: No test for adler32 (adler32-zlib) [ 2819.842325] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2820.076586] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2821.799786] Key type lgssc registered [ 2823.614444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2832.029899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2832.059222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.957321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.933878] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2843.126461] LustreError: 99181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.134910] LustreError: 99181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2846.622409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.995563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2850.867981] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2858.129457] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2863.126568] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2869.871727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2870.217789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.482835] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2878.459276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2883.007734] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2891.745938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2891.756045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.784206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.794958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.800218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.818629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2896.350523] Lustre: server umount lustre-OST0000 complete [ 2903.529081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2903.537262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.554169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.565338] Lustre: Skipped 1 previous similar message [ 2906.361614] Lustre: server umount lustre-MDT0000 complete [ 2909.153937] LustreError: 99182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.165949] LustreError: 99182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2909.442865] LustreError: 99163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396616 with bad export cookie 1435191486120003607 [ 2909.455059] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.456862] LustreError: 99163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2909.750192] Lustre: server umount lustre-MDT0001 complete [ 2920.687621] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2923.417861] Key type lgssc unregistered [ 2923.720542] LNet: 102714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2923.725454] LNetError: 102714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2923.747268] LNet: Removed LNI 192.168.206.155@tcp [ 2924.726194] Key type .llcrypt unregistered [ 2924.729086] Key type ._llcrypt unregistered [ 2941.980383] Key type ._llcrypt registered [ 2941.987890] Key type .llcrypt registered [ 2942.115459] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2943.851900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2943.880213] alg: No test for adler32 (adler32-zlib) [ 2945.099544] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2945.413641] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2947.087164] Key type lgssc registered [ 2948.542620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2955.127798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2955.142237] LustreError: 103796:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2955.155568] LustreError: 103800:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2955.160826] LustreError: 103796:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2955.169607] LustreError: 103796:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2955.178690] LustreError: 103796:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2955.184596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.743577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.364907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.632319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2969.082069] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2974.163474] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing lsmod [ 2977.054770] LustreError: 105004:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2977.062908] LustreError: 105008:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2977.068249] LustreError: 105004:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2977.075555] LustreError: 105004:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2977.086745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2977.091586] LustreError: 105004:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2977.442294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.322457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2980.331069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.340191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.880888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.883358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.889543] Lustre: Skipped 1 previous similar message [ 2982.909927] Lustre: Skipped 1 previous similar message [ 2988.002457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2988.012903] Lustre: Skipped 3 previous similar messages [ 2990.050293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2990.058405] Lustre: Skipped 2 previous similar messages [ 2994.143275] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2995.169753] LustreError: 104384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.205116] LustreError: 104384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2996.348711] Lustre: server umount lustre-MDT0000 complete [ 3000.023132] LustreError: 103816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396706 with bad export cookie 6644978167126388720 [ 3000.042981] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.289077] LustreError: 103833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.294220] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.311237] LustreError: 104269:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003b103ebd x1874221231788928/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3000.328663] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3000.465214] Lustre: server umount lustre-MDT0001 complete [ 3006.542427] Lustre: server umount lustre-OST0000 complete [ 3014.853411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.299684] LustreError: 106071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.411302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3020.083894] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3020.778358] LustreError: 106072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.894649] LustreError: 106071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3029.114758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.488292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3033.684447] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3041.888883] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3047.385534] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3053.887104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3054.169412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.071823] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3060.334641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3066.991383] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3075.900929] Lustre: server umount lustre-OST0000 complete [ 3080.673833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3080.680668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.702175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3089.457210] Lustre: server umount lustre-MDT0000 complete [ 3091.366117] Lustre: 103202:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396782/real 1787396782] req@00000000607cd3ba x1874221231814912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787396798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3091.388104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.396180] Lustre: Skipped 1 previous similar message [ 3091.400416] LustreError: 106072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3091.409575] LustreError: 106072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3092.983276] LustreError: 106053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396799 with bad export cookie 6644978167126389406 [ 3092.988645] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.990052] LustreError: 106053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3093.360909] Lustre: server umount lustre-MDT0001 complete [ 3102.786240] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3105.507672] Key type lgssc unregistered [ 3105.781586] LNet: 109603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3105.790760] LNetError: 109603:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3105.808339] LNet: Removed LNI 192.168.206.155@tcp [ 3106.774989] Key type .llcrypt unregistered [ 3106.784786] Key type ._llcrypt unregistered [ 3128.142144] Key type ._llcrypt registered [ 3128.144554] Key type .llcrypt registered [ 3128.231400] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3129.078457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3129.124633] alg: No test for adler32 (adler32-zlib) [ 3130.257194] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 3130.528299] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3132.223153] Key type lgssc registered [ 3133.162373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3140.014603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3140.057305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.465579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.880561] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3151.844036] LustreError: 111001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.863785] LustreError: 111001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3154.194200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.546852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3159.130950] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3165.939588] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3171.217700] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3177.837904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.185607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.929136] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3186.423307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3190.743660] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3195.455453] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3198.851786] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3205.092650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3205.106396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.124958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3205.437616] Lustre: server umount lustre-OST0000 complete [ 3210.724088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3210.733171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.752365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.265740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.266249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.378524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.393260] Lustre: Skipped 2 previous similar messages [ 3219.503390] Lustre: server umount lustre-MDT0000 complete [ 3222.506495] LustreError: 112290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.527211] LustreError: 112290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3223.232266] LustreError: 110982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396930 with bad export cookie 16639233014421059240 [ 3223.239922] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.247052] LustreError: 110982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3223.610919] Lustre: server umount lustre-MDT0001 complete [ 3234.064425] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3237.161177] Key type lgssc unregistered [ 3237.429639] LNet: 114638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3237.435301] LNetError: 114638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3237.457880] LNet: Removed LNI 192.168.206.155@tcp [ 3238.241926] Key type .llcrypt unregistered [ 3238.243907] Key type ._llcrypt unregistered [ 3254.877971] Key type ._llcrypt registered [ 3254.882291] Key type .llcrypt registered [ 3254.988471] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3256.188706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3256.232112] alg: No test for adler32 (adler32-zlib) [ 3257.294296] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 3257.544941] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3259.231651] Key type lgssc registered [ 3260.286865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3268.443195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3268.462852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.932302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3275.196832] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3279.334357] LustreError: 115859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.356895] LustreError: 115859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3284.411736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3284.448538] LustreError: 115860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.916864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3289.298474] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3296.739707] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3301.906747] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3308.200054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3308.552247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.023507] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3314.737574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3321.200960] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3330.016410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3330.027394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.059828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.149427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.160400] Lustre: Skipped 1 previous similar message [ 3335.424207] Lustre: server umount lustre-OST0000 complete [ 3340.261913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.273109] Lustre: Skipped 1 previous similar message [ 3340.276027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3340.285420] Lustre: Skipped 2 previous similar messages [ 3345.377470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.386308] Lustre: Skipped 1 previous similar message [ 3350.497685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3350.504338] Lustre: Skipped 1 previous similar message [ 3354.080665] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3354.337734] Lustre: server umount lustre-MDT0000 complete [ 3355.617361] LustreError: 118352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.639667] LustreError: 118352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3358.315414] LustreError: 116594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397065 with bad export cookie 15283623655344548753 [ 3358.328319] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.338939] LustreError: 116594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3358.791665] Lustre: server umount lustre-MDT0001 complete [ 3369.064579] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3371.540381] Key type lgssc unregistered [ 3371.953178] LNet: 119393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3371.966970] LNetError: 119393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3372.005833] LNet: Removed LNI 192.168.206.155@tcp [ 3373.136306] Key type .llcrypt unregistered [ 3373.149414] Key type ._llcrypt unregistered [ 3400.226622] Key type ._llcrypt registered [ 3400.228388] Key type .llcrypt registered [ 3400.309711] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3401.110931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3401.140769] alg: No test for adler32 (adler32-zlib) [ 3402.289483] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 3402.581245] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3404.280351] Key type lgssc registered [ 3405.473254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3412.644523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3412.674140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.069168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.293187] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3423.212398] LustreError: 120812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.230710] LustreError: 120812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3426.126753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.453979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3430.894776] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3438.522461] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3444.890363] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3451.756706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3452.221841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.463234] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3458.379427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3466.159918] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3476.894945] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3480.983778] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3480.991858] Lustre: Skipped 2 previous similar messages [ 3485.333430] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3485.362851] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3486.018694] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3486.022985] Lustre: Skipped 2 previous similar messages [ 3495.510853] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3495.518207] Lustre: Skipped 1 previous similar message [ 3501.283651] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3501.294032] Lustre: Skipped 2 previous similar messages [ 3503.137397] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.55@tcp 192.168.0.[2,10]@tcp [ 3507.873476] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.55/24@tcp in log lustre-MDT0000 [ 3507.878777] Lustre: Skipped 2 previous similar messages [ 3511.129447] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.55/24@tcp [ 3511.135239] Lustre: Skipped 1 previous similar message [ 3517.921975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3517.930409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.940794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3517.953786] Lustre: Skipped 1 previous similar message [ 3517.983552] Lustre: Skipped 1 previous similar message [ 3519.756453] Lustre: server umount lustre-OST0000 complete [ 3524.064576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3524.072736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.090621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3528.162545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3528.164388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.898096] Lustre: server umount lustre-MDT0000 complete [ 3533.282945] LustreError: 120811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.293098] LustreError: 123442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397240 with bad export cookie 5806583346699744878 [ 3533.304111] LustreError: 120811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3533.325241] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3533.599932] Lustre: server umount lustre-MDT0001 complete [ 3545.573242] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3549.189553] Key type lgssc unregistered [ 3549.548953] LNet: 125755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3549.569374] LNetError: 125755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3549.589247] LNet: Removed LNI 192.168.206.155@tcp [ 3550.453277] Key type .llcrypt unregistered [ 3550.454809] Key type ._llcrypt unregistered [ 3580.477737] Key type ._llcrypt registered [ 3580.484609] Key type .llcrypt registered [ 3580.587663] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3582.112992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3582.277283] alg: No test for adler32 (adler32-zlib) [ 3583.545335] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 3583.928825] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3585.719181] Key type lgssc registered [ 3587.140323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3595.303256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3595.322779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.822725] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3596.846078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.869867] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.55/24@tcp [ 3602.093960] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3607.009381] LustreError: 127317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3607.032875] LustreError: 127317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3611.237402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.558529] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3611.579134] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.55/24@tcp [ 3611.644134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3616.521700] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3624.179154] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3630.748143] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3638.033588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3638.434520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.524594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3645.417480] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3653.101364] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3665.895569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3665.918399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.964812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.440551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.447899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.451488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3671.908611] Lustre: server umount lustre-OST0000 complete [ 3677.665255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.670539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.688268] Lustre: Skipped 1 previous similar message [ 3677.699336] Lustre: Skipped 2 previous similar messages [ 3682.111385] Lustre: server umount lustre-MDT0000 complete [ 3682.784658] LustreError: 127317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.802829] LustreError: 127317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3685.410707] LustreError: 127299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397392 with bad export cookie 8993860045724820144 [ 3685.412736] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.426160] LustreError: 127299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3685.830760] Lustre: server umount lustre-MDT0001 complete [ 3696.275275] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3699.205691] Key type lgssc unregistered [ 3699.551824] LNet: 130906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3699.571629] LNetError: 130906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3699.590084] LNet: Removed LNI 192.168.206.155@tcp [ 3700.583564] Key type .llcrypt unregistered [ 3700.593906] Key type ._llcrypt unregistered [ 3727.754522] Key type ._llcrypt registered [ 3727.756179] Key type .llcrypt registered [ 3727.870207] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 3742.711480] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3743.648669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3743.658185] alg: No test for adler32 (adler32-zlib) [ 3744.864909] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 3745.030513] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3746.711163] Key type lgssc registered [ 3748.006798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3782.713192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3782.747021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.973474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3783.997306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3784.064467] Lustre: lustre-MDT0000: new disk, initializing [ 3784.143254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.167657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3787.928451] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3797.762710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.858135] Lustre: 134304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3797.930773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3797.941143] Lustre: Skipped 1 previous similar message [ 3798.000605] Lustre: lustre-MDT0001: new disk, initializing [ 3798.098223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3798.152673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3798.162963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3802.095475] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3810.909629] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3816.621595] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3823.598872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3823.914751] Lustre: lustre-OST0000: new disk, initializing [ 3823.918738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3823.931416] Lustre: 136270:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3824.027260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3830.351866] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3830.877460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3830.893923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3831.056519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3839.371664] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3844.027662] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3844.244977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3848.818317] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3849.042389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3851.750990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3851.751844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.758429] LustreError: Skipped 1 previous similar message [ 3851.760780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.777365] Lustre: Skipped 1 previous similar message [ 3856.875769] Lustre: server umount lustre-OST0000 complete [ 3861.984840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.995928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.998106] Lustre: Skipped 1 previous similar message [ 3866.644311] Lustre: server umount lustre-MDT0000 complete [ 3869.616198] LustreError: 134295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397576 with bad export cookie 14621489419052736269 [ 3869.623642] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3869.893815] Lustre: server umount lustre-MDT0001 complete [ 3878.133828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.773333] LustreError: 138489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3878.806588] LustreError: 138489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3879.006483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.441490] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3884.000578] LustreError: 138490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3889.121785] LustreError: 138489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.781570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.089301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3896.420904] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3903.558201] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3908.520433] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3914.779907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3915.012648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.938435] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3927.266354] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3931.411606] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3931.576461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3935.220534] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3935.420314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3943.886104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3944.094481] Lustre: lustre-OST0001: new disk, initializing [ 3944.102889] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3944.112082] Lustre: 141862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3944.236340] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3950.130283] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3950.649180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3950.664828] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3950.792137] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3957.430949] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3957.612439] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3961.474943] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3961.667899] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3966.408450] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98ac8a18b000.ost_server_uuid 50 [ 3967.699740] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98ac8a18b000.ost_server_uuid in IDLE state after 0 sec [ 3976.674591] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3976.676851] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.703219] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3976.722599] Lustre: Skipped 2 previous similar messages [ 3981.399267] Lustre: server umount lustre-OST0001 complete [ 3981.797348] LustreError: 140310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.814602] LustreError: 140310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3985.892655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3985.907264] LustreError: Skipped 1 previous similar message [ 3985.912041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.928246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3985.934989] Lustre: Skipped 1 previous similar message [ 3986.918353] LustreError: 140308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.951586] LustreError: 140308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3992.037128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.046069] Lustre: Skipped 3 previous similar messages [ 3997.160866] LustreError: 141081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3997.178786] LustreError: 141081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3999.204214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3999.493729] Lustre: server umount lustre-OST0000 complete [ 4004.832478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4004.838672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.848571] Lustre: Skipped 1 previous similar message [ 4004.851666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4004.858940] Lustre: Skipped 2 previous similar messages [ 4009.479604] Lustre: server umount lustre-MDT0000 complete [ 4012.784161] LustreError: 138471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397719 with bad export cookie 14621489419052737151 [ 4012.796613] LustreError: 138471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4012.801170] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4013.121945] Lustre: server umount lustre-MDT0001 complete [ 4029.625808] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 4033.035854] Key type lgssc unregistered [ 4033.345538] LNet: 144864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4033.351289] LNetError: 144864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4033.368904] LNet: Removed LNI 192.168.206.155@tcp [ 4034.342744] Key type .llcrypt unregistered [ 4034.349275] Key type ._llcrypt unregistered [ 4071.197031] Key type ._llcrypt registered [ 4071.201138] Key type .llcrypt registered [ 4071.302198] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4083.614273] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4084.562888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4084.616715] alg: No test for adler32 (adler32-zlib) [ 4085.656848] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 4085.889552] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 4087.568273] Key type lgssc registered [ 4088.683653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4121.510468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4121.564270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.854647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4122.887697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4122.976448] Lustre: lustre-MDT0000: new disk, initializing [ 4123.040228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4123.062876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4127.782901] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4138.220206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.360762] Lustre: 148996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4138.407868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4138.410282] Lustre: Skipped 1 previous similar message [ 4138.468768] Lustre: lustre-MDT0001: new disk, initializing [ 4138.541713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4138.584354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4138.598643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4142.837408] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4150.877473] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4156.689482] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4163.270732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4163.477615] Lustre: lustre-OST0000: new disk, initializing [ 4163.480189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4163.489515] Lustre: 150970:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4163.583379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4169.561871] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4172.364545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4172.368993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4172.440086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4178.837827] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4183.862952] Lustre: Failing over lustre-OST0000 [ 4184.102353] Lustre: server umount lustre-OST0000 complete [ 4201.951098] Lustre: 147224:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397892/real 1787397892] req@00000000478386d8 x1874222427177088/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787397908 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4201.977731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.127538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4202.415198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.434886] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4203.039235] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4204.449518] Lustre: 147223:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397895/real 1787397895] req@000000007465b23a x1874222427178112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787397911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4207.975683] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4209.503632] Lustre: 147221:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397900/real 1787397900] req@0000000036442d96 x1874222427179264/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787397916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4209.551082] Lustre: 147221:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4211.610828] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4211.611589] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4211.811768] Lustre: Failing over lustre-MDT0000 [ 4212.038368] Lustre: server umount lustre-MDT0000 complete [ 4214.879117] Lustre: 147221:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397905/real 1787397905] req@000000001e7a5cff x1874222427180288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787397921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4214.913640] Lustre: 147221:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4215.264082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4215.271878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.284479] Lustre: Skipped 1 previous similar message [ 4216.806754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.487206] LustreError: 150088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4218.519458] LustreError: 150088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4221.921407] LustreError: 149009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.951322] LustreError: 149009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4223.599544] LustreError: 149004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.044423] LustreError: 150088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.092660] LustreError: 150088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4228.575131] Lustre: 149000:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397919/real 1787397919] req@00000000d6356a4d x1874222427189504/t0(0) o101->MGC192.168.206.155@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1787397935 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4228.610219] Lustre: 149000:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4228.622308] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4228.644321] Lustre: 149000:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4232.213502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4232.820744] LustreError: 149005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4232.846705] LustreError: 149005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4238.836449] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x55b051815688e294 [ 4238.851091] Lustre: MGC192.168.206.155@tcp: Connection restored to 0@lo (at 0@lo) [ 4238.857218] Lustre: Skipped 1 previous similar message [ 4239.089777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4239.138883] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4243.059703] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4243.998443] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4244.469706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4244.481606] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4244.512057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4252.643709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4252.652426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.661441] Lustre: Skipped 1 previous similar message [ 4252.665131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4254.709731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4258.634628] Lustre: server umount lustre-OST0000 complete [ 4264.929810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4264.930486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.934764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4264.934771] Lustre: Skipped 1 previous similar message [ 4264.966562] Lustre: Skipped 2 previous similar messages [ 4270.052492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4270.057380] Lustre: Skipped 2 previous similar messages [ 4275.169381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4275.174042] Lustre: Skipped 1 previous similar message [ 4276.191195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4276.547894] Lustre: server umount lustre-MDT0000 complete [ 4279.277356] LustreError: 149004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.299469] LustreError: 149004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4280.578100] LustreError: 148989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397987 with bad export cookie 6174524705069392532 [ 4280.594598] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4281.132290] Lustre: server umount lustre-MDT0001 complete [ 4292.062565] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 4294.721841] Key type lgssc unregistered [ 4295.027632] LNet: 154628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4295.032771] LNetError: 154628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4295.043180] LNet: Removed LNI 192.168.206.155@tcp [ 4295.911450] Key type .llcrypt unregistered [ 4295.914106] Key type ._llcrypt unregistered [ 4329.951715] Key type ._llcrypt registered [ 4329.954711] Key type .llcrypt registered [ 4330.046671] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4331.071691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4331.231265] alg: No test for adler32 (adler32-zlib) [ 4332.307807] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 4332.562083] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 4334.263218] Key type lgssc registered [ 4335.419050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4342.898966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4342.925521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.388178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.755393] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4354.529331] LustreError: 156150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.550104] LustreError: 156150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4357.022702] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4359.660030] LustreError: 156149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.670063] LustreError: 156178:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4360.681820] LustreError: 156178:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4360.693838] LustreError: 156178:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4361.082266] Lustre: server umount lustre-MDT0000 complete [ 4374.649686] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4383.648031] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4420.402029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.750257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4420.775471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4420.864715] Lustre: lustre-MDT0000: new disk, initializing [ 4420.979800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.019645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4424.840051] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4434.714273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.840941] Lustre: 159483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4434.923239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4434.930189] Lustre: Skipped 1 previous similar message [ 4435.068919] Lustre: lustre-MDT0001: new disk, initializing [ 4435.198347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4435.236736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4435.244468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4439.528368] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4448.681920] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4453.683472] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4459.284283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4459.548247] Lustre: lustre-OST0000: new disk, initializing [ 4459.556374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4459.561930] Lustre: 161448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4459.671554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.922470] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4466.731363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4466.739559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4466.820687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4472.641326] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4477.132984] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4478.746856] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:29:42 (1787398182) [ 4696.547154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4696.577179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4696.603367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.567914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.570115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.584388] Lustre: Skipped 1 previous similar message [ 4697.610064] Lustre: Skipped 2 previous similar messages [ 4698.901902] Lustre: server umount lustre-MDT0000 complete [ 4702.154042] LustreError: 159473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398409 with bad export cookie 15420790778147176667 [ 4702.158614] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4702.178942] LustreError: 159473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4702.688043] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.690028] LustreError: 159489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.701214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4707.810157] LustreError: 159488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4707.810327] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4708.639744] Lustre: server umount lustre-MDT0001 complete [ 4726.241187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4726.372701] Lustre: server umount lustre-OST0000 complete [ 4737.267947] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4747.838398] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4780.934636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4781.128791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4781.189476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4781.269226] Lustre: lustre-MDT0000: new disk, initializing [ 4781.368987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4781.391606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4785.868759] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4795.951386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.081411] Lustre: 166106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4796.132660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4796.138114] Lustre: Skipped 1 previous similar message [ 4796.215340] Lustre: lustre-MDT0001: new disk, initializing [ 4796.280763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4796.305384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4796.318208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4800.165538] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4808.234659] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4812.987483] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4819.139312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.359635] Lustre: lustre-OST0000: new disk, initializing [ 4819.362692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4819.367942] Lustre: 168075:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4819.454241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.531232] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4827.274033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4827.279597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4827.469699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4834.045595] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4839.586533] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4839.914370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4844.207724] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4844.514931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4848.104317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4848.104428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.116566] LustreError: Skipped 1 previous similar message [ 4848.122245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4848.135102] Lustre: Skipped 1 previous similar message [ 4858.337683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4858.348787] Lustre: Skipped 4 previous similar messages [ 4860.383768] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4860.513478] Lustre: server umount lustre-OST0000 complete [ 4868.064744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4868.077122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4869.984402] Lustre: server umount lustre-MDT0000 complete [ 4873.284512] LustreError: 166099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398580 with bad export cookie 15420790778147335490 [ 4873.292746] LustreError: 166099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4873.293965] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4873.669642] Lustre: server umount lustre-MDT0001 complete [ 4881.165756] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:36:25 (1787398585) [ 4894.825840] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4903.390380] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4934.843096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.165207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4935.195893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4935.307228] Lustre: lustre-MDT0000: new disk, initializing [ 4935.388850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4935.399733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4939.931575] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4950.469461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.598892] Lustre: 172831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4950.841880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4955.009918] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4963.150315] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4968.387981] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4974.431567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4974.705504] Lustre: lustre-OST0000: new disk, initializing [ 4974.708286] Lustre: Skipped 1 previous similar message [ 4974.711592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4974.722151] Lustre: Skipped 2 previous similar messages [ 4974.729695] Lustre: 174809:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4980.279904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4980.295527] Lustre: Skipped 1 previous similar message [ 4980.302207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4980.378850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4980.557476] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4988.602428] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4996.065127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4996.066205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.073653] LustreError: Skipped 1 previous similar message [ 4996.075531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4996.083787] Lustre: Skipped 2 previous similar messages [ 4996.104623] Lustre: Skipped 3 previous similar messages [ 4996.972856] Lustre: server umount lustre-OST0000 complete [ 5002.208703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5007.746347] Lustre: server umount lustre-MDT0000 complete [ 5011.426230] LustreError: 172843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.440692] LustreError: 172843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5012.233024] LustreError: 174808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398719 with bad export cookie 15420790778147336379 [ 5012.238551] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5012.247618] LustreError: 174808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5012.635043] Lustre: server umount lustre-MDT0001 complete [ 5019.695138] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:38:44 (1787398724) [ 5032.440385] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 5040.921692] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5073.399728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.703674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5073.747555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5073.881931] Lustre: lustre-MDT0000: new disk, initializing [ 5073.949975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5073.963090] Lustre: Skipped 2 previous similar messages [ 5073.977134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5078.915223] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5090.323461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5090.494049] Lustre: 179129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5090.773610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5094.893551] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5103.708045] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5109.385538] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5116.257150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5116.495530] Lustre: 181105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5122.818474] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5125.704717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5125.868109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5131.501246] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5141.478953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5141.479233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5141.488955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5141.488962] Lustre: Skipped 3 previous similar messages [ 5141.491734] LustreError: Skipped 1 previous similar message [ 5141.506439] Lustre: Skipped 3 previous similar messages [ 5144.212399] Lustre: server umount lustre-OST0000 complete [ 5153.273168] Lustre: server umount lustre-MDT0000 complete [ 5156.549587] LustreError: 179120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398863 with bad export cookie 15420790778147337464 [ 5156.555114] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5156.558611] LustreError: 179120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5156.845401] Lustre: server umount lustre-MDT0001 complete [ 5167.131249] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5169.447669] Key type lgssc unregistered [ 5169.716346] LNet: 182962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5169.727922] LNetError: 182962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5169.745229] LNet: Removed LNI 192.168.206.155@tcp [ 5170.495285] Key type .llcrypt unregistered [ 5170.498199] Key type ._llcrypt unregistered [ 5189.615294] Key type ._llcrypt registered [ 5189.616937] Key type .llcrypt registered [ 5189.699866] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5190.490423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5190.580900] alg: No test for adler32 (adler32-zlib) [ 5191.661324] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 5191.895708] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5193.567146] Key type lgssc registered [ 5194.662504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5201.408579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5201.428694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.906421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5207.146878] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5212.130797] LustreError: 184355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5212.143839] LustreError: 184355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5215.546846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.898524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5220.288408] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5227.320365] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5232.310196] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5238.705102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5239.113349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.344405] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5247.284843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5252.216457] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5272.037695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5272.044538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.072871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5273.055986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5273.060133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5273.061557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.070516] Lustre: Skipped 1 previous similar message [ 5276.351421] Lustre: server umount lustre-OST0000 complete [ 5282.273588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5282.284907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.313853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5286.625194] Lustre: server umount lustre-MDT0000 complete [ 5288.418585] LustreError: 184355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.444386] LustreError: 184355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5290.144472] LustreError: 184336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398997 with bad export cookie 15501419080904180187 [ 5290.148662] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.154860] LustreError: 184336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5290.386513] Lustre: server umount lustre-MDT0001 complete [ 5300.326461] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5302.943427] Key type lgssc unregistered [ 5303.219760] LNet: 187908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5303.235636] LNetError: 187908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5304.297179] LNet: Removed LNI 192.168.206.155@tcp [ 5305.030302] Key type .llcrypt unregistered [ 5305.035685] Key type ._llcrypt unregistered [ 5330.858651] Key type ._llcrypt registered [ 5330.860789] Key type .llcrypt registered [ 5330.945636] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5331.813107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5331.849331] alg: No test for adler32 (adler32-zlib) [ 5332.811793] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 5333.170749] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5334.839167] Key type lgssc registered [ 5336.111699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5344.162993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5344.204070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5345.763592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5351.029130] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5356.024195] LustreError: 189326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.047753] LustreError: 189326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5359.660479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.094705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5365.190827] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5372.648526] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5377.825782] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5384.164341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5384.505876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.564744] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5390.668198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5399.090812] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5405.840994] Lustre: lustre-OST0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 5406.178909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5406.186426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5406.189992] LustreError: Skipped 1 previous similar message [ 5406.211960] Lustre: Skipped 1 previous similar message [ 5410.932967] Lustre: lustre-OST0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 5410.938932] Lustre: Skipped 2 previous similar messages [ 5411.082206] Lustre: server umount lustre-OST0000 complete [ 5419.376406] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff98ac90b41800.ost_server_uuid 50 [ 5421.194836] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5435.627531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.629851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5435.649575] Lustre: Skipped 1 previous similar message [ 5435.661941] Lustre: Skipped 1 previous similar message [ 5440.993099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5441.001797] Lustre: Skipped 1 previous similar message [ 5441.966201] Lustre: server umount lustre-MDT0000 complete [ 5445.295165] LustreError: 189307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399152 with bad export cookie 3055019502370013689 [ 5445.303844] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5445.306393] LustreError: 189307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5445.592152] Lustre: server umount lustre-MDT0001 complete [ 5454.799092] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:45:58 (1787399158) [ 5462.218667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.795821] LustreError: 193245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5462.883432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5467.497877] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5468.136701] LustreError: 193246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5472.229253] LustreError: 193245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5475.900314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.138149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5481.907313] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5489.591794] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5495.163878] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5502.236794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5502.492222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5507.627130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5507.727619] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5513.773318] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5520.435244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5520.614678] Lustre: lustre-OST0001: new disk, initializing [ 5520.620439] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5520.628431] Lustre: 196087:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5520.736872] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5527.008895] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5530.204667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5530.218940] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5530.313896] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5536.063749] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5540.839243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.855295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5540.863633] Lustre: Skipped 1 previous similar message [ 5540.881264] Lustre: Skipped 1 previous similar message [ 5544.958364] Lustre: server umount lustre-OST0000 complete [ 5545.956124] LustreError: 195069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5545.982447] LustreError: 195069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5551.571754] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5551.785743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5556.180486] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5556.196927] LustreError: 195068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.217500] LustreError: 195068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5556.436580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5571.552844] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5571.567617] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.585206] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5571.599686] Lustre: Skipped 2 previous similar messages [ 5576.625518] Lustre: server umount lustre-OST0001 complete [ 5581.794140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.803554] Lustre: Skipped 2 previous similar messages [ 5586.918608] LustreError: 193245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.957268] LustreError: 193245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5587.037372] Lustre: server umount lustre-MDT0000 complete [ 5590.704200] LustreError: 196086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399297 with bad export cookie 3055019502370014767 [ 5590.713176] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5590.723371] LustreError: 196086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5591.185769] Lustre: server umount lustre-MDT0001 complete [ 5613.666337] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:48:38 (1787399318) [ 5619.521696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.776628] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5619.805812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5620.011891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5624.098746] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5632.262437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.381262] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5632.437433] Lustre: 199523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5638.172793] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5646.467962] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5652.148868] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5659.063415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5659.389213] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5665.464704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5666.276600] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5674.521275] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5681.958179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5682.041177] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5687.505035] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5689.230463] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5696.922444] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5701.091264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.101153] LustreError: Skipped 1 previous similar message [ 5701.114483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.144103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.152443] Lustre: Skipped 2 previous similar messages [ 5705.595976] Lustre: server umount lustre-OST0000 complete [ 5707.887927] LustreError: 201376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.616108] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5723.625039] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5723.628723] LustreError: Skipped 1 previous similar message [ 5723.659213] Lustre: Skipped 2 previous similar messages [ 5728.335279] Lustre: server umount lustre-OST0001 complete [ 5733.863160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5733.881422] Lustre: Skipped 7 previous similar messages [ 5739.724738] Lustre: server umount lustre-MDT0000 complete [ 5743.572793] LustreError: 200280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399450 with bad export cookie 3055019502370016174 [ 5743.581914] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5743.604198] LustreError: 200280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5744.028416] Lustre: server umount lustre-MDT0001 complete [ 5767.148574] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:51:10 (1787399470) [ 5784.861926] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 5794.317192] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5830.477218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.721702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5830.784388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5830.871848] Lustre: lustre-MDT0000: new disk, initializing [ 5830.944763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.947820] Lustre: Skipped 3 previous similar messages [ 5830.967820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5836.150617] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5847.217409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5847.344910] Lustre: 207632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5847.438564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5847.443987] Lustre: Skipped 1 previous similar message [ 5847.506424] Lustre: lustre-MDT0001: new disk, initializing [ 5847.653410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5847.665623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5852.187670] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5861.310423] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5866.767557] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5873.548417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5873.831124] Lustre: lustre-OST0000: new disk, initializing [ 5873.834897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5873.840661] Lustre: 209600:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5880.250880] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5880.399482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5880.409087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5880.533816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5889.127225] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5893.479141] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5893.649297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5897.871026] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5898.133557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5901.280422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5901.290287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.292462] LustreError: Skipped 1 previous similar message [ 5901.295497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5901.303954] Lustre: Skipped 3 previous similar messages [ 5901.322285] Lustre: Skipped 3 previous similar messages [ 5914.080102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5914.240888] Lustre: server umount lustre-OST0000 complete [ 5919.201601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5924.015377] Lustre: server umount lustre-MDT0000 complete [ 5926.888566] LustreError: 207644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.918503] LustreError: 207644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5927.856789] LustreError: 207625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399634 with bad export cookie 3055019502370017959 [ 5927.857954] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5927.866231] LustreError: 207625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5928.253684] Lustre: server umount lustre-MDT0001 complete [ 5937.397247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.115583] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5952.924538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5958.811701] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5966.674621] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5972.125204] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5979.200458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5979.529882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5979.536682] Lustre: Skipped 4 previous similar messages [ 5986.188993] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5993.882344] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5999.007372] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5999.324459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6004.602465] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6004.840263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6008.296464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6008.302832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6008.330662] Lustre: Skipped 2 previous similar messages [ 6013.275099] Lustre: server umount lustre-OST0000 complete [ 6020.122976] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6024.650144] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6053.174929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6058.630091] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6059.563239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.580134] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6064.863925] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6073.038318] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6074.926365] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.946292] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6079.972493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6079.977950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6079.987161] LustreError: Skipped 1 previous similar message [ 6079.997092] Lustre: Skipped 10 previous similar messages [ 6083.859744] Lustre: server umount lustre-OST0000 complete [ 6095.069458] Lustre: server umount lustre-MDT0000 complete [ 6095.334930] LustreError: 214262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6095.352706] LustreError: 214262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6099.007113] LustreError: 211799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399805 with bad export cookie 3055019502370018841 [ 6099.016190] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6099.028731] LustreError: 211799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6099.368929] Lustre: server umount lustre-MDT0001 complete [ 6108.330225] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:56:52 (1787399812) [ 6114.351157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.270960] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6129.141403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6134.833448] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6142.834756] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6149.035565] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6155.609107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6162.262385] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6169.595246] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6174.265550] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6174.553446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6179.659321] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6179.922321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6186.453713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6186.622300] Lustre: lustre-OST0001: new disk, initializing [ 6186.626424] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6186.632706] Lustre: 221158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6192.173600] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6195.290195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6195.295191] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6195.397373] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6201.499246] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6206.180394] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6206.475344] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6210.758716] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6211.098904] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6216.162548] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6216.169356] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6216.197098] Lustre: Skipped 6 previous similar messages [ 6216.228514] LustreError: Skipped 1 previous similar message [ 6218.788635] Lustre: server umount lustre-OST0001 complete [ 6223.585937] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6223.707938] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6226.513537] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6226.660063] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6255.161988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6255.451864] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6255.460633] Lustre: Skipped 5 previous similar messages [ 6260.730908] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6260.742340] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6260.749407] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6260.776064] Lustre: Skipped 1 previous similar message [ 6261.403576] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6268.378477] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6270.430990] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6287.328679] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6287.415160] Lustre: server umount lustre-OST0001 complete [ 6310.437852] Lustre: server umount lustre-OST0000 complete [ 6319.912545] Lustre: server umount lustre-MDT0000 complete [ 6323.247037] LustreError: 217768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400030 with bad export cookie 3055019502370019800 [ 6323.251385] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6323.270812] LustreError: 217768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6329.709254] Lustre: server umount lustre-MDT0001 complete [ 6352.503123] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 08:00:56 (1787400056) [ 6359.828805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.079210] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6360.100557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6365.423496] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6374.325706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.449924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6374.508532] Lustre: 226410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6379.655633] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6387.023638] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6392.105718] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6398.839849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.098917] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6405.207517] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6412.424313] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6421.984256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6422.041703] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6428.806327] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6435.039972] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6436.701194] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6441.703971] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6442.052668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6447.658957] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6448.001571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6455.647725] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98ac90b42800.ost_server_uuid 50 [ 6457.280431] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98ac90b42800.ost_server_uuid in IDLE state after 0 sec [ 6458.680917] Lustre: Permanently deactivating lustre-OST0001 [ 6458.710661] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6462.471746] Lustre: Permanently reactivating lustre-OST0001 [ 6468.064572] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6468.078130] LustreError: Skipped 3 previous similar messages [ 6468.079638] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6468.092118] Lustre: Skipped 20 previous similar messages [ 6472.806559] Lustre: server umount lustre-OST0001 complete [ 6473.192779] LustreError: 228254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.241100] LustreError: 228254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 6478.305697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.327062] Lustre: Skipped 8 previous similar messages [ 6483.350741] Lustre: server umount lustre-OST0000 complete [ 6494.023516] Lustre: server umount lustre-MDT0000 complete [ 6497.647673] LustreError: 226402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400204 with bad export cookie 3055019502370021095 [ 6497.658363] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6497.661353] LustreError: 226402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6498.220131] Lustre: server umount lustre-MDT0001 complete [ 6522.902930] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:03:46 (1787400226) [ 6531.249621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6531.709294] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6531.746743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6531.760770] Lustre: Skipped 5 previous similar messages [ 6536.832695] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6546.190625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.282799] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6546.332773] Lustre: 233105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6552.080256] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6560.138325] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6566.389168] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6573.651345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.856153] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6573.888765] Lustre: Permanently deactivating lustre-OST0000 [ 6573.904755] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6580.268057] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6581.358732] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6590.349584] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6598.269092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6598.386582] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6605.870724] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6607.619303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6614.720909] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6616.423791] Lustre: Permanently reactivating lustre-OST0000 [ 6616.435311] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6616.441039] Lustre: Skipped 2 previous similar messages [ 6624.914520] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6624.915067] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 44 seconds. I think it's dead, and I am evicting it. exp 000000006e4eaf71, cur 1787400331 deadline 1787400317 last 1787400287 [ 6624.925965] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6624.950320] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6625.013356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6636.324829] Lustre: server umount lustre-OST0001 complete [ 6647.099440] Lustre: server umount lustre-OST0000 complete [ 6658.034211] Lustre: server umount lustre-MDT0000 complete [ 6662.469443] LustreError: 233097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400369 with bad export cookie 3055019502370022768 [ 6662.474202] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6662.487120] LustreError: 233097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6663.068645] Lustre: server umount lustre-MDT0001 complete [ 6674.791593] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6677.359393] Key type lgssc unregistered [ 6677.648237] LNet: 237986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6677.658049] LNetError: 237986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6677.674286] LNet: Removed LNI 192.168.206.155@tcp [ 6678.674673] Key type .llcrypt unregistered [ 6678.677293] Key type ._llcrypt unregistered [ 6704.338614] Key type ._llcrypt registered [ 6704.340713] Key type .llcrypt registered [ 6704.455860] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6705.614759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6705.706070] alg: No test for adler32 (adler32-zlib) [ 6707.141990] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 6707.526922] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6709.231191] Key type lgssc registered [ 6711.007070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6724.014924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6724.051519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6725.715949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6731.072568] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6735.856862] LustreError: 239457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6735.896788] LustreError: 239457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6740.969878] LustreError: 239456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.378555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.502118] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6741.543587] Lustre: 239444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6741.556950] Lustre: Permanently deactivating lustre-MDT0001 [ 6741.580666] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6741.855987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6747.013922] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6751.789766] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6755.702678] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6762.120606] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6769.169677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.658979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6770.668861] LustreError: 241299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6770.681956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6775.784809] LustreError: 241299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6776.692498] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6780.906296] LustreError: 241298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.160320] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6791.418674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6791.716842] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6796.782894] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6796.792723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6798.595967] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6806.901339] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6810.908549] Lustre: Permanently reactivating lustre-MDT0001 [ 6810.923420] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6810.932583] Lustre: Skipped 2 previous similar messages [ 6817.439192] Lustre: lustre-MDT0001: Client d805b07b-e371-41bc-a9f0-8f5bd805e69e (at 192.168.206.55@tcp) reconnecting [ 6821.587566] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.618713] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6821.630272] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6821.640822] LustreError: 243168:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6821.647765] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6821.649747] LustreError: 243168:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6821.649857] LustreError: 243168:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.155@tcp: cfg command failed: rc = -108 [ 6821.649878] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6821.649878] [ 6821.650347] LustreError: 239448:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6823.874604] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6825.653568] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6831.948971] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6832.196636] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6837.211286] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6837.443140] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6840.351216] Lustre: Permanently deactivating lustre-MDT0001 [ 6840.371926] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6840.379208] Lustre: Skipped 2 previous similar messages [ 6849.177334] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6857.498350] Lustre: server umount lustre-MDT0000 complete [ 6858.214282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6858.223439] LustreError: 239456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.230581] Lustre: Skipped 3 previous similar messages [ 6858.249064] LustreError: 239456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6861.040624] LustreError: 243941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400567 with bad export cookie 5111070711096290755 [ 6861.046033] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6861.049363] LustreError: 243941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6861.404502] Lustre: server umount lustre-MDT0001 complete [ 6867.715754] Lustre: server umount lustre-OST0000 complete [ 6877.780574] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:09:41 (1787400581) [ 6889.197918] Lustre: server umount lustre-OST0001 complete [ 6896.889838] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 6907.787258] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6946.028627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6946.352051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6946.364980] Lustre: Skipped 2 previous similar messages [ 6946.387760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6946.492207] Lustre: lustre-MDT0000: new disk, initializing [ 6946.566419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6946.582518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6950.986111] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6961.429296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6961.538487] Lustre: 247714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6961.581991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6961.585261] Lustre: Skipped 1 previous similar message [ 6961.643538] Lustre: lustre-MDT0001: new disk, initializing [ 6961.722242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6961.759049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6961.769581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6966.401064] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6974.990873] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6980.719607] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6987.584279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6987.860338] Lustre: lustre-OST0000: new disk, initializing [ 6987.864463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6987.871243] Lustre: 249686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6987.950537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.817298] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6996.006804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6996.016484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6996.093727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7003.184382] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7005.060945] LustreError: 248590:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7013.653247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7013.843324] Lustre: lustre-OST0001: new disk, initializing [ 7013.853534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7013.867649] Lustre: 250802:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7015.127133] LustreError: 248590:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7015.158344] LustreError: 248877:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7019.844795] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7021.196051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7021.207762] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7021.327237] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7025.176144] LustreError: 248877:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7029.971056] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7031.865630] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7036.899264] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7036.906707] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.917613] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7036.923622] Lustre: Skipped 1 previous similar message [ 7037.577205] Lustre: lustre-OST0001: Not available for connect from 192.168.206.55@tcp (stopping) [ 7039.880925] Lustre: server umount lustre-OST0001 complete [ 7042.042085] LustreError: 249668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7042.054527] LustreError: 249668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7057.377457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7057.389030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.405177] LustreError: Skipped 1 previous similar message [ 7057.410211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7057.426063] Lustre: Skipped 2 previous similar messages [ 7057.456335] Lustre: Skipped 1 previous similar message [ 7062.499803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7062.505687] Lustre: Skipped 1 previous similar message [ 7067.619567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7067.625024] Lustre: Skipped 1 previous similar message [ 7071.202141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7071.403580] Lustre: server umount lustre-OST0000 complete [ 7077.857556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.866412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7077.881761] Lustre: Skipped 1 previous similar message [ 7077.906694] Lustre: Skipped 1 previous similar message [ 7081.598340] Lustre: server umount lustre-MDT0000 complete [ 7082.978423] LustreError: 247719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7082.987049] LustreError: 247719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 7085.416614] LustreError: 247707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400792 with bad export cookie 5111070711096293352 [ 7085.419418] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7085.437239] LustreError: 247707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7091.980773] Lustre: server umount lustre-MDT0001 complete [ 7103.452316] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7106.295560] Key type lgssc unregistered [ 7106.647687] LNet: 252836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7106.667418] LNetError: 252836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7106.695680] LNet: Removed LNI 192.168.206.155@tcp [ 7107.835881] Key type .llcrypt unregistered [ 7107.842566] Key type ._llcrypt unregistered [ 7150.622964] Key type ._llcrypt registered [ 7150.626934] Key type .llcrypt registered [ 7150.762754] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7152.325663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7152.362997] alg: No test for adler32 (adler32-zlib) [ 7153.469820] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7153.719488] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7155.391727] Key type lgssc registered [ 7156.735612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7164.289446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7164.305447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.601972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7165.649388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7165.936472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7171.059983] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7180.446106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.525565] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7180.562770] Lustre: 255049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7180.914662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7185.425185] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7193.300745] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7198.703351] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7205.807495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7205.972193] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7206.162302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.557708] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7213.752027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7219.911894] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7274.976434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7274.997837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7275.018679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7275.499697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7275.513883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7275.537110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7275.550932] Lustre: Skipped 1 previous similar message [ 7277.790254] Lustre: server umount lustre-OST0000 complete [ 7282.360599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7296.028330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7296.415336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7300.518305] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7300.540351] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7303.347439] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7310.794459] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7315.957170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7315.973891] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7351.776259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7351.784924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.788468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7351.812125] Lustre: Skipped 1 previous similar message [ 7356.590446] Lustre: server umount lustre-OST0000 complete [ 7362.018859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.022191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7362.031762] Lustre: Skipped 1 previous similar message [ 7362.049416] Lustre: Skipped 2 previous similar messages [ 7366.978719] Lustre: server umount lustre-MDT0000 complete [ 7370.654340] LustreError: 256910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401077 with bad export cookie 8424536823467573734 [ 7370.658072] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.663083] LustreError: 256910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7371.119941] Lustre: server umount lustre-MDT0001 complete [ 7383.898503] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7387.193891] Key type lgssc unregistered [ 7387.498825] LNet: 260471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7387.508618] LNetError: 260471:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7387.528155] LNet: Removed LNI 192.168.206.155@tcp [ 7388.454251] Key type .llcrypt unregistered [ 7388.458560] Key type ._llcrypt unregistered [ 7414.007853] Key type ._llcrypt registered [ 7414.009713] Key type .llcrypt registered [ 7414.104877] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7415.790045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7415.814172] alg: No test for adler32 (adler32-zlib) [ 7417.016508] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7417.329698] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7419.031677] Key type lgssc registered [ 7420.213677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7428.020570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7428.061643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.722063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7434.481038] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7439.846041] LustreError: 261891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7439.872021] LustreError: 261891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7443.570786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.990734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7449.066291] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7456.692351] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7462.227808] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7468.914865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7469.204021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7474.363951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7476.104547] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7483.532489] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7509.478965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7509.479483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7509.493540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7509.516213] Lustre: Skipped 1 previous similar message [ 7514.600667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7514.604389] Lustre: Skipped 2 previous similar messages [ 7514.965501] Lustre: server umount lustre-OST0000 complete [ 7519.717053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7519.729390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7519.740379] Lustre: Skipped 1 previous similar message [ 7524.417536] Lustre: server umount lustre-MDT0000 complete [ 7524.834482] LustreError: 261891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.844064] LustreError: 261891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7527.325965] LustreError: 261872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401234 with bad export cookie 4815247314835559064 [ 7527.329239] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.336481] LustreError: 261872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7527.552315] Lustre: server umount lustre-MDT0001 complete [ 7538.024841] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7539.965507] Key type lgssc unregistered [ 7540.281314] LNet: 266195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7540.285960] LNetError: 266195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7541.356938] LNet: Removed LNI 192.168.206.155@tcp [ 7542.105635] Key type .llcrypt unregistered [ 7542.109521] Key type ._llcrypt unregistered [ 7561.070819] Key type ._llcrypt registered [ 7561.078173] Key type .llcrypt registered [ 7561.228509] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7562.725778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7562.818271] alg: No test for adler32 (adler32-zlib) [ 7563.798144] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7563.973420] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7565.607201] Key type lgssc registered [ 7567.040286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7574.567544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7574.594045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.056810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7580.680490] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7585.248746] LustreError: 267419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.281506] LustreError: 267419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7589.179556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.485099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7594.081422] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7600.361666] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7604.891538] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7610.617858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7610.866818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.789753] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7621.779894] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7622.130580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7632.359188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7632.368263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.398071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7637.351848] Lustre: server umount lustre-OST0000 complete [ 7642.600299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7642.611504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7642.613466] Lustre: Skipped 2 previous similar messages [ 7642.627433] Lustre: Skipped 2 previous similar messages [ 7647.712779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7647.722446] Lustre: Skipped 1 previous similar message [ 7652.833506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7652.836987] Lustre: Skipped 1 previous similar message [ 7655.391231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7655.818209] Lustre: server umount lustre-MDT0000 complete [ 7657.953219] LustreError: 267420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7657.978877] LustreError: 267420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7659.728925] LustreError: 267402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401366 with bad export cookie 6683397791632947493 [ 7659.735632] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.743187] LustreError: 267402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7660.015790] Lustre: server umount lustre-MDT0001 complete [ 7671.356190] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7674.917275] Key type lgssc unregistered [ 7675.243579] LNet: 271097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7675.247312] LNetError: 271097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7676.327984] LNet: Removed LNI 192.168.206.155@tcp [ 7677.620070] Key type .llcrypt unregistered [ 7677.647122] Key type ._llcrypt unregistered [ 7702.805292] Key type ._llcrypt registered [ 7702.807427] Key type .llcrypt registered [ 7702.926373] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7704.140435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7704.169646] alg: No test for adler32 (adler32-zlib) [ 7705.204392] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7705.443293] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7707.107337] Key type lgssc registered [ 7708.110218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7715.575885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7715.587418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7717.040545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7721.247372] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7726.567348] LustreError: 272513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7726.588868] LustreError: 272513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7728.815733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.204132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7733.865861] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7740.729959] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7746.381248] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7752.859935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7753.277635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7760.189568] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7761.469775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7766.983381] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7790.050184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7790.056651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.076165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7792.096420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7792.102910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.103994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7796.120320] Lustre: server umount lustre-OST0000 complete [ 7800.800816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7800.810905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.828070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7800.831431] Lustre: Skipped 1 previous similar message [ 7805.665771] Lustre: server umount lustre-MDT0000 complete [ 7807.457447] LustreError: 272512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7807.471581] LustreError: 272512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7809.036845] LustreError: 272494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401515 with bad export cookie 513498138788916460 [ 7809.042496] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7809.051133] LustreError: 272494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7809.372724] Lustre: server umount lustre-MDT0001 complete [ 7820.574977] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7823.294876] Key type lgssc unregistered [ 7823.542762] LNet: 276818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7823.549586] LNetError: 276818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7823.567756] LNet: Removed LNI 192.168.206.155@tcp [ 7824.509335] Key type .llcrypt unregistered [ 7824.512624] Key type ._llcrypt unregistered [ 7843.619412] Key type ._llcrypt registered [ 7843.620818] Key type .llcrypt registered [ 7843.785396] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7844.914853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7844.929819] alg: No test for adler32 (adler32-zlib) [ 7845.998307] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7846.274705] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7847.927604] Key type lgssc registered [ 7849.113698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7857.015123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7857.046735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7858.527632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.344637] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7868.907384] LustreError: 278054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7868.926272] LustreError: 278054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7871.842467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.306981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7877.444206] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7884.459950] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7890.444908] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7897.364366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7897.782493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.920872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7904.198128] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7911.627077] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7924.709689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7924.711842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.714057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7924.725669] LustreError: Skipped 1 previous similar message [ 7924.754109] Lustre: Skipped 1 previous similar message [ 7929.534476] Lustre: server umount lustre-OST0000 complete [ 7934.202367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.211516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7934.221485] Lustre: Skipped 1 previous similar message [ 7934.242068] Lustre: Skipped 2 previous similar messages [ 7939.552131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7939.559032] Lustre: Skipped 1 previous similar message [ 7940.651230] Lustre: server umount lustre-MDT0000 complete [ 7944.542666] LustreError: 278019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401651 with bad export cookie 12092344801497570085 [ 7944.549227] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7944.557238] LustreError: 278019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7945.006895] Lustre: server umount lustre-MDT0001 complete [ 7957.763344] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7961.184919] Key type lgssc unregistered [ 7961.473098] LNet: 281754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7961.481458] LNetError: 281754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7961.515850] LNet: Removed LNI 192.168.206.155@tcp [ 7962.450611] Key type .llcrypt unregistered [ 7962.453374] Key type ._llcrypt unregistered [ 7974.174809] Key type ._llcrypt registered [ 7974.176911] Key type .llcrypt registered [ 7974.297618] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7992.820780] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8009.725822] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8011.460473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8011.486043] alg: No test for adler32 (adler32-zlib) [ 8012.694038] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 8013.009966] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8014.631300] Key type lgssc registered [ 8015.785948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8054.184656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8054.201112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.485746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8055.528633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8055.661438] Lustre: lustre-MDT0000: new disk, initializing [ 8055.783919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8055.818711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8060.583379] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8072.148334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.293847] Lustre: 285314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8072.372880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8072.378753] Lustre: Skipped 1 previous similar message [ 8072.502371] Lustre: lustre-MDT0001: new disk, initializing [ 8072.621722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8072.678543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8072.689193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8077.545487] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8087.377281] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8092.649547] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8099.896049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8100.152667] Lustre: lustre-OST0000: new disk, initializing [ 8100.156626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8100.163984] Lustre: 287282:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8100.311959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8107.582603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8107.594544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8107.744893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8107.874625] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8118.339697] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8122.983586] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8123.266369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8128.378680] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8128.635773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8133.600746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8133.610074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.617478] LustreError: Skipped 1 previous similar message [ 8133.637610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8138.725899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8138.736892] Lustre: Skipped 2 previous similar messages [ 8143.851614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8143.861413] Lustre: Skipped 1 previous similar message [ 8145.375233] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8145.657337] Lustre: server umount lustre-OST0000 complete [ 8154.080872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.083213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8154.099468] Lustre: Skipped 2 previous similar messages [ 8154.115655] Lustre: Skipped 1 previous similar message [ 8159.200616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8159.207406] Lustre: Skipped 1 previous similar message [ 8164.319168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8164.656885] Lustre: server umount lustre-MDT0000 complete [ 8168.421032] LustreError: 285307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401875 with bad export cookie 11730509687464888893 [ 8168.442038] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8168.915307] Lustre: server umount lustre-MDT0001 complete [ 8177.533566] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:31:21 (1787401881) [ 8184.015565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.652075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.922120] LustreError: 289698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8189.955936] LustreError: 289698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8190.186490] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8195.044980] LustreError: 289697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8198.914701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.280529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8204.814712] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8212.980455] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8220.880058] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8228.601759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8229.162648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8235.659984] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8242.354599] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8254.957845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.960018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8254.982113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8254.992266] Lustre: Skipped 2 previous similar messages [ 8260.007917] Lustre: server umount lustre-OST0000 complete [ 8265.188200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8265.220862] Lustre: Skipped 1 previous similar message [ 8270.226216] Lustre: server umount lustre-MDT0000 complete [ 8270.304282] LustreError: 289692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.337131] LustreError: 289692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8274.202349] LustreError: 289678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401981 with bad export cookie 11730509687464889775 [ 8274.209865] LustreError: 289678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8274.210447] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8274.540918] Lustre: server umount lustre-MDT0001 complete [ 8288.112315] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8290.917364] Key type lgssc unregistered [ 8291.297945] LNet: 293235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8291.303781] LNetError: 293235:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8291.321545] LNet: Removed LNI 192.168.206.155@tcp [ 8292.372548] Key type .llcrypt unregistered [ 8292.375194] Key type ._llcrypt unregistered [ 8319.159718] Key type ._llcrypt registered [ 8319.163962] Key type .llcrypt registered [ 8319.305713] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8334.679686] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8335.813715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8335.845902] alg: No test for adler32 (adler32-zlib) [ 8336.888588] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 8337.200607] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8338.967173] Key type lgssc registered [ 8340.061930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8384.264143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8384.320047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.491930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8385.510294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8385.587680] Lustre: lustre-MDT0000: new disk, initializing [ 8385.663181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8385.675408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8389.769617] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8399.247090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8399.347148] Lustre: 297061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8399.412148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8399.416674] Lustre: Skipped 1 previous similar message [ 8399.494590] Lustre: lustre-MDT0001: new disk, initializing [ 8399.606632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8399.642866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8399.661240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8404.049080] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8413.804234] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8418.950053] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8425.268472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8425.501891] Lustre: lustre-OST2710: new disk, initializing [ 8425.508208] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8425.513429] Lustre: 299031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8425.594219] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8431.616166] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8435.291037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8435.298649] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8435.449407] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8441.203388] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8448.150580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8448.388605] Lustre: lustre-OST03e8: new disk, initializing [ 8448.395758] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8448.404715] Lustre: 300093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8448.527908] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8455.499716] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8455.770323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8455.786173] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8455.924038] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8462.763807] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8470.680147] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8470.951885] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8476.325487] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8476.606603] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8481.283120] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8481.562305] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8485.862943] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8486.124953] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8492.010521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.020990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8492.028484] Lustre: Skipped 3 previous similar messages [ 8492.049558] Lustre: Skipped 3 previous similar messages [ 8492.492488] Lustre: server umount lustre-MDT0000 complete [ 8496.438103] LustreError: 297056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402203 with bad export cookie 10713611298475863076 [ 8496.446488] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8496.450758] LustreError: 297056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8496.997908] Lustre: server umount lustre-MDT0001 complete [ 8511.656124] Lustre: server umount lustre-OST2710 complete [ 8513.567948] Lustre: 294861:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402204/real 1787402204] req@00000000c26a316a x1874226885755264/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8513.614782] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.766396] Lustre: server umount lustre-OST03e8 complete [ 8533.898437] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8544.229612] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8577.784172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8578.008806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8578.044310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8578.102515] Lustre: lustre-MDT0000: new disk, initializing [ 8578.213734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8578.242224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8584.065331] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8594.906655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8594.989279] Lustre: 305340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8595.027866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8595.030606] Lustre: Skipped 1 previous similar message [ 8595.109554] Lustre: lustre-MDT0001: new disk, initializing [ 8595.181895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8595.229401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8595.242701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8599.599330] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8609.167453] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8614.539297] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8621.083555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8621.276557] Lustre: lustre-OST0000: new disk, initializing [ 8621.280526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8621.287831] Lustre: 307312:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8621.370515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8627.513892] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8630.362773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8630.375500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8630.515843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8637.422090] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8642.498914] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8642.675437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8647.530411] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8647.710458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8651.234027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8651.249134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.278853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8655.899282] Lustre: server umount lustre-OST0000 complete [ 8661.477621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8661.499583] Lustre: Skipped 1 previous similar message [ 8661.504276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8661.514067] Lustre: Skipped 2 previous similar messages [ 8665.996432] Lustre: server umount lustre-MDT0000 complete [ 8670.009311] LustreError: 305333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402376 with bad export cookie 10713611298475864756 [ 8670.031211] LustreError: 305333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8670.032607] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8670.384393] Lustre: server umount lustre-MDT0001 complete [ 8679.499516] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:39:43 (1787402383) [ 8681.112496] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8683.139367] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:39:47 (1787402387) [ 8686.839708] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8714.235655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8714.523781] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8714.548583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8714.716468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.320868] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8728.347826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.442477] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8728.484248] Lustre: 311168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8728.555535] LustreError: 311180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8728.579429] LustreError: 311180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8733.265928] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8754.733686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8755.073302] Lustre: Denying initial registration attempt from nid 192.168.206.155@tcp, specified as failover [ 8755.087693] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8755.124453] LustreError: 312621:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8755.133747] LustreError: 312621:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8755.145233] LustreError: 312621:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8755.251741] Lustre: server umount lustre-OST0000 complete [ 8755.261103] LustreError: 312621:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8764.898322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.916924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8764.924480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8764.924488] LustreError: Skipped 1 previous similar message [ 8764.931769] Lustre: Skipped 2 previous similar messages [ 8764.966523] Lustre: Skipped 1 previous similar message [ 8769.238177] Lustre: server umount lustre-MDT0000 complete [ 8770.017913] LustreError: 311175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8770.042948] LustreError: 311175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8773.228307] LustreError: 311163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402480 with bad export cookie 10713611298475865638 [ 8773.232625] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8773.238868] LustreError: 311163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8773.800650] Lustre: server umount lustre-MDT0001 complete [ 8782.479100] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8785.652189] Key type lgssc unregistered [ 8785.987650] LNet: 313743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8785.996389] LNetError: 313743:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8786.022982] LNet: Removed LNI 192.168.206.155@tcp [ 8786.864203] Key type .llcrypt unregistered [ 8786.866568] Key type ._llcrypt unregistered [ 8798.711519] Key type ._llcrypt registered [ 8798.714509] Key type .llcrypt registered [ 8798.809774] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8799.656916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8799.820509] alg: No test for adler32 (adler32-zlib) [ 8800.919892] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 8801.209537] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8802.943319] Key type lgssc registered [ 8804.004954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8849.783983] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8860.194914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8860.256808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.515532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8861.528239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8861.749677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.605099] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8874.699577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.881814] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8874.916477] Lustre: 316508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8875.276641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8879.711911] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8886.694071] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8892.412855] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8911.594421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8911.800541] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8912.006230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8917.764798] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8924.373078] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8927.201806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8927.208212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8927.223976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8927.720379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8927.724703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8927.738195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8928.124440] Lustre: server umount lustre-OST0000 complete [ 8931.811621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8931.824324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8931.840789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8937.288971] Lustre: server umount lustre-MDT0000 complete [ 8940.226292] LustreError: 316501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402647 with bad export cookie 6695842479384941256 [ 8940.227337] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8940.232809] LustreError: 316501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8940.479676] Lustre: server umount lustre-MDT0001 complete [ 8950.873368] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8953.458397] Key type lgssc unregistered [ 8953.760714] LNet: 320082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8953.779280] LNetError: 320082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8953.808625] LNet: Removed LNI 192.168.206.155@tcp [ 8954.718959] Key type .llcrypt unregistered [ 8954.721692] Key type ._llcrypt unregistered [ 8977.471169] Key type ._llcrypt registered [ 8977.474098] Key type .llcrypt registered [ 8977.564840] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8978.258024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8978.297087] alg: No test for adler32 (adler32-zlib) [ 8979.350854] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 8979.597463] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8981.247366] Key type lgssc registered [ 8982.392250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8988.435236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8988.452451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.798287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8993.096446] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8999.357299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8999.394354] LustreError: 321501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.416654] LustreError: 321501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8999.690286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9003.415203] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9009.084558] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9013.667306] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9021.357023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9021.731977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9026.321584] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9032.381382] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9044.660109] Lustre: server umount lustre-MDT0000 complete [ 9045.481230] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9045.484266] LustreError: 321501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9045.499496] Lustre: Skipped 1 previous similar message [ 9045.529286] LustreError: 321501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9048.088507] LustreError: 321482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402754 with bad export cookie 7704625574110802020 [ 9048.102725] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9049.232524] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [ 9049.232552] LustreError: 321496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9050.391828] Lustre: server umount lustre-MDT0001 complete [ 9056.477853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9064.276521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.975622] Lustre: 320812:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402757/real 1787402757] req@00000000d280346a x1874227558902912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9067.010259] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9067.019385] Lustre: Skipped 1 previous similar message [ 9070.047728] Lustre: 320810:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402760/real 1787402760] req@00000000234ce494 x1874227558903040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9071.135283] Lustre: 320810:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402762/real 1787402762] req@000000002e35836e x1874227558903168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9073.183847] LustreError: 320808:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d755bc1e x1874227558904064/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9073.538807] LustreError: 323330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9073.604292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9073.653499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9074.593639] Lustre: 320811:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402765/real 1787402765] req@00000000899c91bf x1874227558903424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402781 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9077.795471] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9079.775237] Lustre: 320812:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402770/real 1787402770] req@000000002364dd81 x1874227558903552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402786 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9081.825980] LustreError: 325110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9081.844265] LustreError: 325110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9084.741076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.047503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9086.071563] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9088.807953] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9095.663593] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9097.238872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9102.193769] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9103.784774] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9109.472498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9109.478200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9109.489380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9110.496524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9110.512179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9117.158034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9117.166572] Lustre: Skipped 3 previous similar messages [ 9122.273345] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9122.304774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9122.316067] Lustre: Skipped 1 previous similar message [ 9122.410587] Lustre: server umount lustre-OST0000 complete [ 9125.857820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9125.864368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.873050] Lustre: Skipped 1 previous similar message [ 9132.514358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9132.525258] Lustre: Skipped 4 previous similar messages [ 9140.191337] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9140.429549] Lustre: server umount lustre-MDT0000 complete [ 9142.753940] LustreError: 325114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9142.783974] LustreError: 325114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9143.769574] LustreError: 321480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402850 with bad export cookie 7704625574110809160 [ 9143.780837] LustreError: 321480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9143.783617] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9144.068608] Lustre: server umount lustre-MDT0001 complete [ 9154.909875] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9158.236494] Key type lgssc unregistered [ 9158.599796] LNet: 327788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9158.605627] LNetError: 327788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9158.630770] LNet: Removed LNI 192.168.206.155@tcp [ 9159.754793] Key type .llcrypt unregistered [ 9159.759802] Key type ._llcrypt unregistered [ 9188.251236] Key type ._llcrypt registered [ 9188.252987] Key type .llcrypt registered [ 9188.382638] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9189.321571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9189.370501] alg: No test for adler32 (adler32-zlib) [ 9190.470761] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 9190.684905] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 9192.367149] Key type lgssc registered [ 9193.864620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9202.185550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9202.213598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9203.733301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9208.196150] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9213.928433] LustreError: 329205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9213.947466] LustreError: 329205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9215.453460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9215.781848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9219.658232] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9226.105363] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9231.490944] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9236.448818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9236.462590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.474771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9238.505522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9238.510499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9240.454712] Lustre: server umount lustre-MDT0000 complete [ 9240.609105] LustreError: 329933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.617431] LustreError: 329933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9244.365510] LustreError: 329184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402951 with bad export cookie 13675417677666116706 [ 9244.368780] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9244.372642] LustreError: 329184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9244.746229] Lustre: server umount lustre-MDT0001 complete [ 9260.847773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9261.098391] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9261.158128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9261.351441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9266.691756] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9275.342608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.454564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9275.478825] Lustre: 331795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9275.720802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9280.439856] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9287.962699] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9293.506662] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9302.704430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9302.945989] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9302.959498] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9302.973206] LustreError: 333678:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9302.978718] LustreError: 333678:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9302.985266] LustreError: 333678:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9303.084340] Lustre: server umount lustre-OST0000 complete [ 9303.087045] LustreError: 333678:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9313.205209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9313.474625] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9313.664398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9319.145979] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9322.631445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9325.480401] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9332.617924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9332.704138] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9332.837799] Lustre: lustre-OST0001: new disk, initializing [ 9332.846967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9332.856106] Lustre: 335219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9332.959073] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9338.510571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9338.531420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9338.619353] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9339.646322] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9347.752603] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9354.237659] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9354.247350] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.284894] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9354.291038] Lustre: Skipped 2 previous similar messages [ 9357.186797] Lustre: server umount lustre-OST0001 complete [ 9359.332742] LustreError: 335518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9359.354982] LustreError: 335518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9363.425711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9363.438731] LustreError: Skipped 1 previous similar message [ 9363.452896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9363.481023] Lustre: Skipped 1 previous similar message [ 9363.501553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9364.456731] LustreError: 335518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9364.493936] LustreError: 335518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9366.811556] Lustre: server umount lustre-OST0000 complete [ 9373.151918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9373.165419] LustreError: Skipped 1 previous similar message [ 9373.175289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9373.192404] Lustre: Skipped 1 previous similar message [ 9373.210900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9373.220821] Lustre: Skipped 3 previous similar messages [ 9379.719370] Lustre: server umount lustre-MDT0000 complete [ 9379.809043] LustreError: 331802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9379.825044] LustreError: 331802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9383.168738] LustreError: 331786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403090 with bad export cookie 13675417677666117259 [ 9383.169754] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9383.186805] LustreError: 331786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9383.698547] Lustre: server umount lustre-MDT0001 complete [ 9394.861346] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9397.800700] Key type lgssc unregistered [ 9398.148719] LNet: 337179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9398.155101] LNetError: 337179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9398.168420] LNet: Removed LNI 192.168.206.155@tcp [ 9399.032378] Key type .llcrypt unregistered [ 9399.034168] Key type ._llcrypt unregistered [ 9467.338729] Key type ._llcrypt registered [ 9467.345068] Key type .llcrypt registered [ 9467.485796] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 9482.422553] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9483.801625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9484.008744] alg: No test for adler32 (adler32-zlib) [ 9485.017965] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 9485.371483] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 9487.095906] Key type lgssc registered [ 9488.180882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9523.713613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9523.745507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9525.002719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9525.052554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9525.182616] Lustre: lustre-MDT0000: new disk, initializing [ 9525.281770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9525.311885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9529.197270] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9540.329892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9540.462694] Lustre: 342116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9540.530691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9540.536349] Lustre: Skipped 1 previous similar message [ 9540.650342] Lustre: lustre-MDT0001: new disk, initializing [ 9540.800809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9540.884036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9540.898770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9546.332439] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9556.740487] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9562.045571] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9568.133358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9568.402189] Lustre: lustre-OST0000: new disk, initializing [ 9568.413743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9568.422478] Lustre: 344084:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9568.483132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9575.447355] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9577.030895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9577.045702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9577.197087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9585.382617] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9591.258791] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9591.499507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9596.841466] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9597.077246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9603.053627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9603.072565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9603.074320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9603.080860] Lustre: Skipped 1 previous similar message [ 9603.108950] LustreError: Skipped 1 previous similar message [ 9608.161698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9608.173498] Lustre: Skipped 2 previous similar messages [ 9613.279148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9613.297586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9613.307727] Lustre: Skipped 1 previous similar message [ 9613.488436] Lustre: server umount lustre-OST0000 complete [ 9617.890689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9617.906517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9617.923607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9623.522607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9623.528475] Lustre: Skipped 3 previous similar messages [ 9623.730795] Lustre: server umount lustre-MDT0000 complete [ 9627.611138] LustreError: 342109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403334 with bad export cookie 18194786576399470107 [ 9627.619080] LustreError: 342109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9627.627249] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9627.937229] Lustre: server umount lustre-MDT0001 complete [ 9637.238600] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:55:40 (1787403340) [ 9646.188587] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:55:49 (1787403349) [ 9656.246920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.554071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9661.206268] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9661.922054] LustreError: 346840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9661.941367] LustreError: 346840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9667.044342] LustreError: 346839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9669.188973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9669.692319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9674.305621] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9681.995201] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9688.780505] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9696.075638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9696.334181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9703.459271] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9711.531924] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9723.722308] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9725.751781] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9727.623214] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9729.773053] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9731.861486] Lustre: Failing over lustre-MDT0000 [ 9732.387707] Lustre: server umount lustre-MDT0000 complete [ 9734.790115] LustreError: 348560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9735.651376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9735.682640] Lustre: Skipped 3 previous similar messages [ 9739.887153] LustreError: 348560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.915129] LustreError: 348560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9745.007432] LustreError: 346834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9745.044454] LustreError: 346834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9751.958651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9752.031306] Lustre: 340342:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403442/real 1787403442] req@0000000051abb8b1 x1874228089528960/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787403458 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9752.067649] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9756.270505] LustreError: 346836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9756.288396] LustreError: 346836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 9761.271029] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfc80ddfdba5615a0 [ 9761.285673] Lustre: MGC192.168.206.155@tcp: Connection restored to 0@lo (at 0@lo) [ 9761.400811] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (not set up) [ 9761.734511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9761.836192] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9766.514268] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9766.910394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9766.967434] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9767.034785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9767.112364] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9777.433970] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9779.303746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9783.574869] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9786.285774] Lustre: Failing over lustre-MDT0000 [ 9786.659636] Lustre: server umount lustre-MDT0000 complete [ 9787.043378] LustreError: 346834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9787.086591] LustreError: 346834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9787.362693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.363440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9787.408533] Lustre: Skipped 2 previous similar messages [ 9803.683065] Lustre: 340340:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403494/real 1787403494] req@000000007f6a18f3 x1874228089549824/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787403510 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9803.699467] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9806.976193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9814.006311] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfc80ddfdba5618b7 [ 9814.022824] Lustre: MGC192.168.206.155@tcp: Connection restored to 0@lo (at 0@lo) [ 9814.036646] Lustre: Skipped 2 previous similar messages [ 9814.416522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9814.466569] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9817.722727] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9819.427425] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9819.632899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9819.675353] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9819.715128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9829.408743] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9831.519369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9840.107013] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9840.116201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9840.123477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9840.138034] Lustre: Skipped 2 previous similar messages [ 9840.153677] Lustre: Skipped 2 previous similar messages [ 9845.201724] Lustre: server umount lustre-MDT0000 complete [ 9845.231564] LustreError: 348560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.262818] LustreError: 348560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 29 previous similar messages [ 9848.654883] LustreError: 346821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403555 with bad export cookie 18194786576399472823 [ 9848.658133] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9848.678974] LustreError: 346821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9849.193663] Lustre: server umount lustre-MDT0001 complete [ 9864.266128] Lustre: server umount lustre-OST0000 complete [ 9881.385207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9881.919121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9886.769435] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9895.884422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9900.677872] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9907.309541] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9912.822307] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9920.151679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9926.622741] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9926.661450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9933.728967] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9941.994073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9941.997941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9942.003706] LustreError: Skipped 1 previous similar message [ 9942.005667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9942.034250] Lustre: Skipped 1 previous similar message [ 9944.007275] Lustre: server umount lustre-OST0000 complete [ 9953.650923] Lustre: server umount lustre-MDT0000 complete [ 9955.298239] LustreError: 353208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9955.324185] LustreError: 353208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9957.553426] LustreError: 353188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403664 with bad export cookie 18194786576399473565 [ 9957.556262] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9957.574043] LustreError: 353188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9957.856279] Lustre: server umount lustre-MDT0001 complete [ 9968.361535] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9971.002616] Key type lgssc unregistered [ 9971.314692] LNet: 356750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9971.325683] LNetError: 356750:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9971.341538] LNet: Removed LNI 192.168.206.155@tcp [ 9972.131278] Key type .llcrypt unregistered [ 9972.134432] Key type ._llcrypt unregistered [10000.193289] Key type ._llcrypt registered [10000.196877] Key type .llcrypt registered [10000.335637] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10001.387001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10001.631725] alg: No test for adler32 (adler32-zlib) [10002.888804] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [10003.240493] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10004.919155] Key type lgssc registered [10006.353166] Lustre: Echo OBD driver; http://www.lustre.org/ [10016.566356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10016.597329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10018.436913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10022.910563] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10029.030200] LustreError: 358265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10029.056304] LustreError: 358265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10031.148687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10031.418019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10035.754069] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10042.553415] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10047.404668] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10054.547298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10054.892064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10061.605485] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10064.120069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10069.000704] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10079.720125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.724496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10079.726815] Lustre: Skipped 2 previous similar messages [10079.734320] Lustre: Skipped 1 previous similar message [10084.835689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10084.842744] Lustre: Skipped 2 previous similar messages [10085.102754] Lustre: server umount lustre-MDT0000 complete [10088.987843] LustreError: 358245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403795 with bad export cookie 15546630249433864448 [10088.990584] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10088.995907] LustreError: 358245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10089.214352] Lustre: server umount lustre-MDT0001 complete [10102.434406] Lustre: server umount lustre-OST0000 complete [10117.078766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10117.453442] LustreError: 361978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.530403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10122.470427] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10122.727499] LustreError: 361979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10127.863103] LustreError: 361978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10131.817310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10132.256292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10137.070083] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10144.974479] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10150.337200] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10156.704247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10157.090245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10162.635714] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10163.242231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10169.776885] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10183.651664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10183.660111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10183.682826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10183.699739] Lustre: Skipped 1 previous similar message [10187.447510] Lustre: server umount lustre-MDT0000 complete [10188.780599] LustreError: 363819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.803892] LustreError: 363819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10191.363203] LustreError: 361959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403898 with bad export cookie 15546630249433865554 [10191.366727] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10191.371518] LustreError: 361959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10191.761543] Lustre: server umount lustre-MDT0001 complete [10206.066711] Lustre: server umount lustre-OST0000 complete [10220.642525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10220.999576] LustreError: 365887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.082226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10225.233930] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10234.105430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.033994] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10246.519194] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10252.707825] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10258.579899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10259.005040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10259.011955] Lustre: Skipped 1 previous similar message [10265.384341] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10266.175718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10272.833662] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10280.225295] Lustre: server umount lustre-OST0000 complete [10286.047991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10286.060865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.081044] Lustre: Skipped 1 previous similar message [10286.089836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10286.107438] Lustre: Skipped 2 previous similar messages [10291.682097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10291.685970] Lustre: Skipped 2 previous similar messages [10294.874752] Lustre: server umount lustre-MDT0000 complete [10296.735109] Lustre: 357577:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403987/real 1787403987] req@000000003f3e5b4d x1874228632705664/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787404003 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10296.761615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10296.776618] Lustre: Skipped 1 previous similar message [10296.785621] LustreError: 365887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10296.813025] LustreError: 365887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10297.824304] Lustre: 357577:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403988/real 1787403988] req@0000000087440b63 x1874228632706688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787404004 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10298.287924] LustreError: 367731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404005 with bad export cookie 15546630249433866583 [10298.288505] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10298.294685] LustreError: 367731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10298.571133] Lustre: server umount lustre-MDT0001 complete [10308.759343] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10311.320932] Key type lgssc unregistered [10311.584933] LNet: 369420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10311.594648] LNetError: 369420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10311.604654] LNet: Removed LNI 192.168.206.155@tcp [10312.309497] Key type .llcrypt unregistered [10312.314201] Key type ._llcrypt unregistered [10333.769512] Key type ._llcrypt registered [10333.772349] Key type .llcrypt registered [10333.873918] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10335.180368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10335.201258] alg: No test for adler32 (adler32-zlib) [10336.281588] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [10336.573371] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10338.263833] Key type lgssc registered [10339.239492] Lustre: Echo OBD driver; http://www.lustre.org/ [10345.607581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10345.619655] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10345.624478] LustreError: 370859:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10345.636308] LustreError: 370859:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10345.642432] LustreError: 370859:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10345.648533] LustreError: 370859:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10345.656390] LustreError: 370859:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10350.991683] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10350.995649] LustreError: 371256:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10351.023364] LustreError: 371256:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10351.029876] LustreError: 371256:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10351.039929] LustreError: 371256:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10351.046786] LustreError: 371256:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10365.951954] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10368.433200] Key type lgssc unregistered [10368.691255] LNet: 372258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10368.697544] LNetError: 372258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10368.712521] LNet: Removed LNI 192.168.206.155@tcp [10369.385091] Key type .llcrypt unregistered [10369.390250] Key type ._llcrypt unregistered [10385.249723] Key type ._llcrypt registered [10385.252267] Key type .llcrypt registered [10385.396744] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [10399.218571] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10400.206342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10400.247097] alg: No test for adler32 (adler32-zlib) [10401.281935] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [10401.497558] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10403.144513] Key type lgssc registered [10404.154474] Lustre: Echo OBD driver; http://www.lustre.org/ [10433.578175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10433.609471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10434.887120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10434.923982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10435.015729] Lustre: lustre-MDT0000: new disk, initializing [10435.089728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10435.113739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10438.769230] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10448.216558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10448.358298] Lustre: 375415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10448.425706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10448.431495] Lustre: Skipped 1 previous similar message [10448.504522] Lustre: lustre-MDT0001: new disk, initializing [10448.595174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10448.639551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10448.649213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10452.436821] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10460.183208] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10464.392071] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10469.504088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10469.776111] Lustre: lustre-OST0000: new disk, initializing [10469.781175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10469.788163] Lustre: 377383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10469.874234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10474.903940] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10479.663330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10479.670943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10479.795187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10483.157624] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10487.264171] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10487.464834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10491.642731] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10491.884095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10494.944990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10494.954609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.970876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10495.461871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10495.462069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10495.479897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10495.502568] Lustre: Skipped 1 previous similar message [10499.828450] Lustre: server umount lustre-OST0000 complete [10505.185635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10505.190159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10505.199350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10509.516624] Lustre: server umount lustre-MDT0000 complete [10512.836246] LustreError: 375408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404219 with bad export cookie 15834350161971754942 [10512.846592] LustreError: 375408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10512.848725] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10513.198549] Lustre: server umount lustre-MDT0001 complete [10520.304535] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10521.679162] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:10:26 (1787404226) [10527.098341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.479871] LustreError: 379841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10527.488833] LustreError: 379841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10527.542655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10531.413126] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10532.843066] LustreError: 379842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10537.957513] LustreError: 379841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10537.978657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10538.275461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10542.379994] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10549.967670] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10555.352455] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10561.546851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.886343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10567.657566] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10574.445758] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10580.577929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10580.796223] Lustre: lustre-OST0001: new disk, initializing [10580.802588] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10580.815882] Lustre: 382674:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10580.944846] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10587.295730] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10589.245828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10589.258895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10589.333954] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10595.702689] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10599.911332] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10599.913592] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10599.930061] LustreError: Skipped 1 previous similar message [10599.935751] Lustre: Skipped 1 previous similar message [10599.939064] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10599.955166] Lustre: Skipped 2 previous similar messages [10605.033358] LustreError: 381660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10605.058261] LustreError: 381660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10605.189798] Lustre: server umount lustre-OST0001 complete [10610.155358] LustreError: 381662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.168292] LustreError: 381662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10620.388347] LustreError: 381660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.419325] LustreError: 381660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10623.457649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10623.463067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.470029] Lustre: Skipped 1 previous similar message [10623.480815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10623.490665] Lustre: Skipped 1 previous similar message [10627.799735] Lustre: server umount lustre-OST0000 complete [10635.748223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10635.753456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10635.756102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10635.756109] Lustre: Skipped 2 previous similar messages [10635.765270] LustreError: Skipped 1 previous similar message [10635.811217] Lustre: Skipped 2 previous similar messages [10638.144746] Lustre: server umount lustre-MDT0000 complete [10640.865119] LustreError: 382681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10640.893702] LustreError: 382681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10642.820466] LustreError: 379823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404349 with bad export cookie 15834350161971755824 [10642.835181] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.838023] LustreError: 379823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10657.247279] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10657.547236] Lustre: server umount lustre-MDT0001 complete [10668.054953] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10670.972376] Key type lgssc unregistered [10671.345439] LNet: 384670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10671.357421] LNetError: 384670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10671.388357] LNet: Removed LNI 192.168.206.155@tcp [10672.479203] Key type .llcrypt unregistered [10672.481545] Key type ._llcrypt unregistered [10722.369453] Key type ._llcrypt registered [10722.372464] Key type .llcrypt registered [10722.490486] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10723.841891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10723.856064] alg: No test for adler32 (adler32-zlib) [10724.894766] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [10725.155043] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10726.817164] Key type lgssc registered [10728.036857] Lustre: Echo OBD driver; http://www.lustre.org/ [10735.824235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10735.846539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10737.130157] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10737.144805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10737.335851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10742.385221] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10751.342846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.520938] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10751.560206] Lustre: 387471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10751.914181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10756.196862] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10762.564356] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10767.162734] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10770.915529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10770.919312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10770.929136] Lustre: Skipped 1 previous similar message [10770.943916] Lustre: Skipped 1 previous similar message [10776.037763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.054579] Lustre: Skipped 1 previous similar message [10781.155453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10781.165544] Lustre: Skipped 1 previous similar message [10783.199312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10783.459106] Lustre: server umount lustre-MDT0000 complete [10787.135792] LustreError: 387463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404494 with bad export cookie 14381451126341406134 [10787.143476] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10787.535967] Lustre: server umount lustre-MDT0001 complete [10796.315950] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:15:00 (1787404500) [10803.207300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.535854] LustreError: 389983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10803.550845] LustreError: 389983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10803.595230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10807.635208] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10808.801372] LustreError: 389984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.926152] LustreError: 389983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.468833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10814.759950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10818.237228] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10824.705607] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10829.717976] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10835.929197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10836.078273] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10836.234893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10842.275914] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10849.570853] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10857.637988] Lustre: Permanently deactivating lustre-OST0000 [10857.656974] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10866.709224] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10868.719881] LustreError: 393016:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10868.726984] LustreError: 393016:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10868.732334] LustreError: 393016:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10871.865534] LustreError: 393064:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10871.872793] LustreError: 393064:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10880.020936] Lustre: server umount lustre-OST0000 complete [10882.555474] LustreError: 393314:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10882.561210] LustreError: 393314:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10886.624625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10886.634435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10886.645056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10888.674308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10893.792346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10893.800211] Lustre: Skipped 2 previous similar messages [10894.891420] Lustre: server umount lustre-MDT0000 complete [10898.580371] LustreError: 389963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404605 with bad export cookie 14381451126341406715 [10898.590586] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10898.597231] LustreError: 389963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10908.473865] Lustre: server umount lustre-MDT0001 complete [10917.339681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10923.197273] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10925.883656] LustreError: 394474:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10932.371630] Lustre: 394813:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.155@tcp [10934.525946] Lustre: server umount MGS complete [10945.315314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10945.853303] LustreError: 395537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.868583] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10945.895506] Lustre: Skipped 1 previous similar message [10945.965929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10950.407411] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10951.137390] LustreError: 395538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10958.491478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10958.724919] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10962.403282] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10968.539907] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10973.632624] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10975.437869] Lustre: Permanently reactivating lustre-OST0000 [10975.448957] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10975.455170] Lustre: Skipped 2 previous similar messages [11002.849038] LustreError: 397318:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11008.941038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11009.267182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11009.277309] Lustre: Skipped 1 previous similar message [11014.395781] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11019.797491] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11027.947644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11027.958578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11027.991434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11028.001257] Lustre: Skipped 1 previous similar message [11030.646552] Lustre: server umount lustre-OST0000 complete [11035.617146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11035.632306] LustreError: Skipped 1 previous similar message [11035.642468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11035.655449] Lustre: Skipped 1 previous similar message [11040.461995] Lustre: server umount lustre-MDT0000 complete [11043.302525] LustreError: 395537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11043.318590] LustreError: 395537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11044.031474] LustreError: 395518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404750 with bad export cookie 14381451126341407765 [11044.036938] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11044.057863] LustreError: 395518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11044.487833] Lustre: server umount lustre-MDT0001 complete [11054.986848] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11057.586287] Key type lgssc unregistered [11057.843666] LNet: 399899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11057.855476] LNetError: 399899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11057.875270] LNet: Removed LNI 192.168.206.155@tcp [11058.698390] Key type .llcrypt unregistered [11058.700437] Key type ._llcrypt unregistered [11076.126349] Key type ._llcrypt registered [11076.129810] Key type .llcrypt registered [11076.231741] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [11090.576517] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11091.467405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11091.593615] alg: No test for adler32 (adler32-zlib) [11092.703908] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11092.932521] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11094.639600] Key type lgssc registered [11096.092969] Lustre: Echo OBD driver; http://www.lustre.org/ [11128.095546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11128.120723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11129.362796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11129.402861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11129.494455] Lustre: lustre-MDT0000: new disk, initializing [11129.629555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11129.654956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11133.991634] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11143.868327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11144.034237] Lustre: 403058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11144.130713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11144.135176] Lustre: Skipped 1 previous similar message [11144.222682] Lustre: lustre-MDT0001: new disk, initializing [11144.333396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11144.370265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11144.388092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11148.417443] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11156.839951] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11162.288197] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11168.870957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11169.130275] Lustre: lustre-OST0000: new disk, initializing [11169.135607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11169.140870] Lustre: 405029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11169.229520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11174.514793] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11174.941141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11174.953118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11175.104932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11183.153813] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11186.549831] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11186.710335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11190.512863] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11190.748892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11195.873733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11195.878689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11195.896483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11195.905156] Lustre: Skipped 1 previous similar message [11198.937777] Lustre: server umount lustre-OST0000 complete [11205.601062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11205.610089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11205.629071] Lustre: Skipped 1 previous similar message [11205.637866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11208.956400] Lustre: server umount lustre-MDT0000 complete [11211.813962] LustreError: 403050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404918 with bad export cookie 15826364668915278704 [11211.817970] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11211.822447] LustreError: 403050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11212.171743] Lustre: server umount lustre-MDT0001 complete [11219.363235] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11221.028380] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:22:05 (1787404925) [11227.022099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11227.397414] LustreError: 407488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11227.413714] LustreError: 407488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11227.462984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11230.903667] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11232.740813] LustreError: 407489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11237.860334] LustreError: 407488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11238.065296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11238.438168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11243.427899] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11250.282590] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11255.393759] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11261.113123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11261.365251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11266.757129] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11273.688156] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11275.386568] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11278.869102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11283.936434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11283.940347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11283.952777] Lustre: Skipped 1 previous similar message [11283.958455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11283.960886] Lustre: Skipped 2 previous similar messages [11288.032801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11288.039542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.063070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11290.113340] Lustre: server umount lustre-OST0000 complete [11294.691260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11294.703686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11294.717829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11294.727872] Lustre: Skipped 1 previous similar message [11300.210430] Lustre: server umount lustre-MDT0000 complete [11303.394792] LustreError: 407485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.417537] LustreError: 407485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11303.989797] LustreError: 407470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405010 with bad export cookie 15826364668915279586 [11303.996162] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11303.999227] LustreError: 407470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11304.385898] Lustre: server umount lustre-MDT0001 complete [11315.344791] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11318.474842] Key type lgssc unregistered [11318.865567] LNet: 411213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11318.878422] LNetError: 411213:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11318.899702] LNet: Removed LNI 192.168.206.155@tcp [11319.843581] Key type .llcrypt unregistered [11319.845493] Key type ._llcrypt unregistered [11342.707086] Key type ._llcrypt registered [11342.711789] Key type .llcrypt registered [11342.812683] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11343.152755] Key type .llcrypt unregistered [11343.158870] Key type ._llcrypt unregistered [11359.401759] Key type ._llcrypt registered [11359.404335] Key type .llcrypt registered [11359.498360] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11360.751937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11360.782066] alg: No test for adler32 (adler32-zlib) [11361.884973] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11362.141270] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11363.799837] Key type lgssc registered [11364.944775] Lustre: Echo OBD driver; http://www.lustre.org/ [11372.614334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11372.646405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11374.201776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11378.710821] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11384.803908] LustreError: 413423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.821928] LustreError: 413423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11387.729849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11388.089477] LustreError: 413424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11388.111737] LustreError: 413424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11388.119889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11393.504476] LustreError: 413423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11393.525618] LustreError: 413423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11394.411484] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11398.629637] LustreError: 413419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11398.652468] LustreError: 413419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11399.664509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11401.040043] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11404.770240] LustreError: 413424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11404.795970] LustreError: 413424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11406.651264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11406.984377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11411.284597] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11420.131220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11420.135530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11420.144315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11422.178806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11422.189132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11422.200281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11424.316326] Lustre: server umount lustre-OST0000 complete [11427.811386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11427.832681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11427.840572] Lustre: Skipped 1 previous similar message [11432.416077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11432.424793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11432.438166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11434.116838] Lustre: server umount lustre-MDT0000 complete [11437.296384] LustreError: 413404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405144 with bad export cookie 6737740613301762605 [11437.299616] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11437.309335] LustreError: 413404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11437.563798] Lustre: server umount lustre-MDT0001 complete [11447.777902] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11450.213517] Key type lgssc unregistered [11450.441709] LNet: 416571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11450.449142] LNetError: 416571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11450.466747] LNet: Removed LNI 192.168.206.155@tcp [11451.233323] Key type .llcrypt unregistered [11451.235228] Key type ._llcrypt unregistered [11470.761886] Key type ._llcrypt registered [11470.764739] Key type .llcrypt registered [11470.874831] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11471.547504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11471.610584] alg: No test for adler32 (adler32-zlib) [11472.594940] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11472.784877] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11474.440326] Key type lgssc registered [11475.466264] Lustre: Echo OBD driver; http://www.lustre.org/ [11481.896494] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11481.921733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11497.439723] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11502.559297] LustreError: 417940:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [11529.183206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11534.344504] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11542.225956] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11548.820617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11548.914162] LustreError: 417275:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f51e5e3e x1874230172977408/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11549.318799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11549.392162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11553.870499] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11554.800219] LustreError: 417966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11554.819947] LustreError: 417966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11559.904894] LustreError: 417965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11561.979215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11562.262786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11566.716341] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11572.850606] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11577.230311] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11583.977531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11583.980756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11583.997598] Lustre: Skipped 1 previous similar message [11585.587249] Lustre: server umount lustre-OST0000 complete [11591.143079] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11591.161467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11591.199558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11591.210149] Lustre: Skipped 1 previous similar message [11591.708917] Lustre: server umount lustre-MDT0000 complete [11594.214288] LustreError: 418981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11594.216474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11596.025981] LustreError: 420466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405302 with bad export cookie 2669212567366013247 [11596.027706] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11596.031815] LustreError: 420466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11596.338708] Lustre: server umount lustre-MDT0001 complete [11607.659357] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11610.336605] Key type lgssc unregistered [11610.527525] LNet: 421505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11610.546516] LNetError: 421505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11610.573602] LNet: Removed LNI 192.168.206.155@tcp [11611.337431] Key type .llcrypt unregistered [11611.339540] Key type ._llcrypt unregistered [11633.150771] Key type ._llcrypt registered [11633.154741] Key type .llcrypt registered [11633.230581] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11634.254515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11634.345682] alg: No test for adler32 (adler32-zlib) [11635.376364] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11635.644817] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11637.311524] Key type lgssc registered [11638.258890] Lustre: Echo OBD driver; http://www.lustre.org/ [11644.932754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11644.945828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.397925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11649.931719] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11656.673610] LustreError: 422930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11656.687542] LustreError: 422930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11657.226351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11657.618209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11661.198765] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11668.034833] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11673.428884] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11679.416470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11679.810966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11685.411527] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11691.756397] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11692.151279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11702.416482] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [11702.755962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11702.773442] Lustre: Skipped 2 previous similar messages [11705.062901] Lustre: server umount lustre-MDT0000 complete [11706.940873] LustreError: 422909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.55@tcp arrived at 1787405413 with bad export cookie 1498034975631967287 [11707.877340] LustreError: 422926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11707.916447] LustreError: 422926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11708.623279] LustreError: 422911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405415 with bad export cookie 1498034975631966622 [11708.635208] LustreError: 422911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11708.644074] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11708.896388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11708.903753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11708.923106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11708.929740] Lustre: Skipped 3 previous similar messages [11712.994297] LustreError: 422930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11713.018763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11713.019599] LustreError: 422930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11714.833861] Lustre: server umount lustre-OST0000 complete [11716.069717] LustreError: 422925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11716.087757] LustreError: 422925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11721.381844] Lustre: server umount lustre-MDT0001 complete [11730.119888] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11731.783520] Key type lgssc unregistered [11732.015706] LNet: 426613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11732.030613] LNetError: 426613:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11732.044482] LNet: Removed LNI 192.168.206.155@tcp [11732.892420] Key type .llcrypt unregistered [11732.897895] Key type ._llcrypt unregistered [11750.560233] Key type ._llcrypt registered [11750.564883] Key type .llcrypt registered [11750.646156] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11751.601485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11751.644618] alg: No test for adler32 (adler32-zlib) [11752.658757] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11752.949607] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11754.615173] Key type lgssc registered [11755.521522] Lustre: Echo OBD driver; http://www.lustre.org/ [11761.837939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11761.860911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11763.240994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11766.700436] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11773.415068] LustreError: 428008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11773.437201] LustreError: 428008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11773.566405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11773.848644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11777.262858] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11783.756669] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11788.523447] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11793.417039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11793.710820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11798.464538] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11804.578708] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11809.073581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11814.375983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11814.392317] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11814.401369] Lustre: Skipped 1 previous similar message [11815.565555] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [11818.744755] Lustre: server umount lustre-MDT0001 complete [11819.490528] LustreError: 428008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11819.514827] LustreError: 428008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11824.610569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11824.611566] LustreError: 429103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11824.631299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11824.631308] Lustre: Skipped 1 previous similar message [11824.639371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11829.728088] LustreError: 428007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11829.728257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11829.926399] Lustre: server umount lustre-OST0000 complete [11833.649526] Lustre: server umount lustre-MDT0000 complete [11845.687375] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11848.161334] Key type lgssc unregistered [11848.417697] LNet: 431687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11848.428187] LNetError: 431687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11848.454777] LNet: Removed LNI 192.168.206.155@tcp [11849.263557] Key type .llcrypt unregistered [11849.266692] Key type ._llcrypt unregistered [11870.045472] Key type ._llcrypt registered [11870.049992] Key type .llcrypt registered [11870.189537] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [11882.124493] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11882.998382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11883.040507] alg: No test for adler32 (adler32-zlib) [11884.038899] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [11884.208320] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11885.864067] Key type lgssc registered [11886.717103] Lustre: Echo OBD driver; http://www.lustre.org/ [11920.663961] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11933.831269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11933.882787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11935.060930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11935.079354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11935.149675] Lustre: lustre-MDT0000: new disk, initializing [11935.213923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11935.232461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11939.098696] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11951.803419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11952.099251] Lustre: lustre-OST0000: new disk, initializing [11952.104236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11952.109325] Lustre: Skipped 1 previous similar message [11952.114601] Lustre: 436697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11952.183444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11957.344421] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11958.339231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11958.347537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11958.380234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11966.508384] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11980.440395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.568841] Lustre: 435680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11980.637583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11980.725966] Lustre: lustre-MDT0001: new disk, initializing [11980.833573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11980.886181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11980.895919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11984.751586] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11993.163349] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11993.370167] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11997.495677] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11997.692788] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12005.346398] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12005.350209] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12005.356467] Lustre: Skipped 1 previous similar message [12005.363444] Lustre: Skipped 1 previous similar message [12007.914574] Lustre: server umount lustre-MDT0001 complete [12015.584863] LustreError: 435687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12015.604850] LustreError: 435687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12020.709411] LustreError: 435689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.728444] LustreError: 435689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12025.827916] LustreError: 435692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12025.842242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12025.871184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12025.880443] Lustre: Skipped 1 previous similar message [12030.946000] LustreError: 435693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12030.946791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12036.065108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12036.071185] LustreError: 435692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12039.135925] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12039.393798] Lustre: server umount lustre-OST0000 complete [12049.180969] Lustre: server umount lustre-MDT0000 complete [12060.495562] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12062.698310] Key type lgssc unregistered [12062.943087] LNet: 440443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12062.950039] LNetError: 440443:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12062.974191] LNet: Removed LNI 192.168.206.155@tcp [12063.700321] Key type .llcrypt unregistered [12063.704389] Key type ._llcrypt unregistered [12097.867099] Key type ._llcrypt registered [12097.873285] Key type .llcrypt registered [12097.956076] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12098.158326] Key type .llcrypt unregistered [12098.161533] Key type ._llcrypt unregistered [12111.257295] Key type ._llcrypt registered [12111.260994] Key type .llcrypt registered [12111.365946] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [12112.316379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12112.447651] alg: No test for adler32 (adler32-zlib) [12113.480659] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [12113.686300] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [12115.343233] Key type lgssc registered [12116.353602] Lustre: Echo OBD driver; http://www.lustre.org/ [12139.206580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12139.242408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12140.552805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12140.580595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12140.654781] Lustre: lustre-MDT0000: new disk, initializing [12140.745488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12140.768219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12144.497546] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12153.363783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12153.495048] Lustre: 443800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12153.543669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12153.549595] Lustre: Skipped 1 previous similar message [12153.631406] Lustre: lustre-MDT0001: new disk, initializing [12153.701183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12153.728426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12153.734108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12157.079659] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12165.301630] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12169.950416] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12175.659488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12175.898932] Lustre: lustre-OST0000: new disk, initializing [12175.902404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12175.910044] Lustre: 445768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12175.988926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12181.399699] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12182.574125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12182.584636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12182.693415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12189.727295] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12198.374477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12198.380675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12198.400946] Lustre: Skipped 1 previous similar message [12198.417717] Lustre: Skipped 1 previous similar message [12201.527345] Lustre: server umount lustre-MDT0000 complete [12205.416266] LustreError: 444596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405912 with bad export cookie 9575844044451568695 [12205.418510] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12205.430988] LustreError: 444596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12205.894898] Lustre: server umount lustre-MDT0001 complete [12219.983859] Lustre: server umount lustre-OST0000 complete [12229.060621] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:38:53 (1787405933) [12235.847428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12236.327933] LustreError: 447796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12236.350040] LustreError: 447796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12236.445763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12240.457543] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12241.904854] LustreError: 447797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.021447] LustreError: 447796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12248.237306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12248.618653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12252.995529] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12260.063966] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12265.265367] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12284.659192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12285.036623] Lustre: Found index 0 for lustre-OST0000, updating log [12285.323619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12293.623938] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12301.709707] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12307.849687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12320.229202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12320.242336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12320.261418] Lustre: Skipped 1 previous similar message [12320.280837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12321.312212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12321.329769] Lustre: Skipped 2 previous similar messages [12324.592591] Lustre: server umount lustre-OST0000 complete [12340.119147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12340.137594] Lustre: Skipped 1 previous similar message [12340.150170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12340.158331] Lustre: Skipped 1 previous similar message [12345.313301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12345.320158] Lustre: Skipped 1 previous similar message [12346.497197] Lustre: server umount lustre-MDT0000 complete [12347.376821] LustreError: 447797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12349.937461] LustreError: 447776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406056 with bad export cookie 9575844044451570088 [12349.952376] LustreError: 447776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12349.955260] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12350.401778] Lustre: server umount lustre-MDT0001 complete [12351.263239] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp [12359.771491] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:41:03 (1787406063) [12973.556519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12973.849932] LustreError: 454230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12973.863276] LustreError: 454230:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12973.908047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12977.804726] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12979.187437] LustreError: 454231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12983.266419] LustreError: 454230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12985.698081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.007193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12989.877985] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12995.395734] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13000.526287] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13019.252186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13019.483497] Lustre: Found index 0 for lustre-OST0000, updating log [13019.798422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13021.889454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13025.667959] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13030.175307] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp [13032.765925] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13035.617419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13035.635821] Lustre: Skipped 1 previous similar message [13035.648890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13040.860912] Lustre: server umount lustre-OST0000 complete [13044.034319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13044.053221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13044.055510] Lustre: Skipped 1 previous similar message [13046.114084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13046.116473] Lustre: Skipped 2 previous similar messages [13050.209715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13050.222214] Lustre: Skipped 7 previous similar messages [13050.361225] Lustre: server umount lustre-MDT0000 complete [13051.236190] LustreError: 454225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.263563] LustreError: 454225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13053.837637] LustreError: 454211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406760 with bad export cookie 9575844044451571180 [13053.840922] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13053.848205] LustreError: 454211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [13054.171371] Lustre: server umount lustre-MDT0001 complete [13085.023507] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [13094.687379] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp [13095.563375] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13129.971424] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 09:53:54 (1787406834) [13146.922537] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [13149.855934] Key type lgssc unregistered [13150.264769] LNet: 460842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13150.272385] LNetError: 460842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13280.358069] LNet: Removed LNI 192.168.206.155@tcp [13281.201209] Key type .llcrypt unregistered [13281.203210] Key type ._llcrypt unregistered [13296.014192] Key type ._llcrypt registered [13296.019207] Key type .llcrypt registered [13296.109064] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13297.023590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13297.147773] alg: No test for adler32 (adler32-zlib) [13298.185128] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [13298.387439] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13300.023181] Key type lgssc registered [13301.156178] Lustre: Echo OBD driver; http://www.lustre.org/ [13305.147269] LNet: Added LNI 192.168.206.155@tcp100 [8/256/0/180] [13305.971670] LNet: Added LNI 192.168.206.155@tcp101 [8/256/0/180] [13306.663445] LNet: Added LNI 192.168.206.155@tcp102 [8/256/0/180] [13307.423072] LNet: Added LNI 192.168.206.155@tcp103 [8/256/0/180] [13308.218851] LNet: Added LNI 192.168.206.155@tcp104 [8/256/0/180] [13308.992709] LNet: Added LNI 192.168.206.155@tcp105 [8/256/0/180] [13309.864884] LNet: Added LNI 192.168.206.155@tcp106 [8/256/0/180] [13310.799332] LNet: Added LNI 192.168.206.155@tcp107 [8/256/0/180] [13311.529862] LNet: Added LNI 192.168.206.155@tcp108 [8/256/0/180] [13312.421279] LNet: Added LNI 192.168.206.155@tcp109 [8/256/0/180] [13313.192493] LNet: Added LNI 192.168.206.155@tcp110 [8/256/0/180] [13313.968389] LNet: Added LNI 192.168.206.155@tcp111 [8/256/0/180] [13314.791636] LNet: Added LNI 192.168.206.155@tcp112 [8/256/0/180] [13315.563539] LNet: Added LNI 192.168.206.155@tcp113 [8/256/0/180] [13316.385630] LNet: Added LNI 192.168.206.155@tcp114 [8/256/0/180] [13317.388876] LNet: Added LNI 192.168.206.155@tcp115 [8/256/0/180] [13318.366769] LNet: Added LNI 192.168.206.155@tcp116 [8/256/0/180] [13319.120872] LNet: Added LNI 192.168.206.155@tcp117 [8/256/0/180] [13319.991337] LNet: Added LNI 192.168.206.155@tcp118 [8/256/0/180] [13320.860185] LNet: Added LNI 192.168.206.155@tcp119 [8/256/0/180] [13321.736975] LNet: Added LNI 192.168.206.155@tcp120 [8/256/0/180] [13322.518954] LNet: Added LNI 192.168.206.155@tcp121 [8/256/0/180] [13323.262265] LNet: Added LNI 192.168.206.155@tcp122 [8/256/0/180] [13324.092379] LNet: Added LNI 192.168.206.155@tcp123 [8/256/0/180] [13324.839698] LNet: Added LNI 192.168.206.155@tcp124 [8/256/0/180] [13325.548386] LNet: Added LNI 192.168.206.155@tcp125 [8/256/0/180] [13326.388807] LNet: Added LNI 192.168.206.155@tcp126 [8/256/0/180] [13327.201818] LNet: Added LNI 192.168.206.155@tcp127 [8/256/0/180] [13327.966560] LNet: Added LNI 192.168.206.155@tcp128 [8/256/0/180] [13328.663760] LNet: Added LNI 192.168.206.155@tcp129 [8/256/0/180] [13329.343907] LNet: Added LNI 192.168.206.155@tcp130 [8/256/0/180] [13330.139827] LNet: Added LNI 192.168.206.155@tcp131 [8/256/0/180] [13330.901559] LNet: Added LNI 192.168.206.155@tcp132 [8/256/0/180] [13331.641980] LNet: Added LNI 192.168.206.155@tcp133 [8/256/0/180] [13332.359341] LNet: Added LNI 192.168.206.155@tcp134 [8/256/0/180] [13333.123459] LNet: Added LNI 192.168.206.155@tcp135 [8/256/0/180] [13339.065400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13339.076444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13340.163430] Lustre: 463874:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13340.261887] LNetError: 463895:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13340.279527] LNetError: Protocol error connecting to 192.168.206.155@tcp118 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13340.285363] LNetError: Skipped 4 previous similar messages [13340.335483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13340.379771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13340.383158] LNet: 7 local NIs in recovery (showing 7): 192.168.206.155@tcp123, 192.168.206.155@tcp121, 192.168.206.155@tcp124, 192.168.206.155@tcp125, 192.168.206.155@tcp126, 192.168.206.155@tcp127, 192.168.206.155@tcp129 [13340.492984] Lustre: lustre-MDT0000: new disk, initializing [13340.573601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13340.596044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13341.344329] LNetError: 463905:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13341.356522] LNetError: 463905:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [13341.369455] LNetError: Protocol error connecting to 192.168.206.155@tcp101 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13341.384098] LNetError: Skipped 5 previous similar messages [13343.470457] LNetError: 463899:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13343.476899] LNetError: 463899:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13343.482281] LNetError: Protocol error connecting to 192.168.206.155@tcp124 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13343.487780] LNetError: Skipped 9 previous similar messages [13344.350774] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13347.616482] LNetError: 461335:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13347.627891] LNetError: 461335:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [13347.634126] LNetError: Protocol error connecting to 192.168.206.155@tcp124 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13347.650423] LNetError: Skipped 2 previous similar messages [13353.504741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13353.638690] Lustre: 463892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13353.706479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13353.712854] Lustre: Skipped 1 previous similar message [13353.783396] Lustre: lustre-MDT0001: new disk, initializing [13353.827719] Lustre: 464696:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13353.837726] Lustre: 464696:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13353.950108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13353.975273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13353.988701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13355.808781] LNetError: 461336:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13355.819242] LNetError: 461336:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [13355.828152] LNetError: Protocol error connecting to 192.168.206.155@tcp124 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13357.142648] Lustre: 464918:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13357.149348] Lustre: 464918:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13357.431584] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13365.368776] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13371.104790] LNetError: 461333:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13371.122772] LNetError: Protocol error connecting to 192.168.206.155@tcp128 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13371.142779] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13378.060362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13378.445654] Lustre: lustre-OST0000: new disk, initializing [13378.458790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13378.473520] Lustre: 465867:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13378.585789] Lustre: 465872:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13378.591454] Lustre: 465872:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13378.636215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13384.221178] Lustre: 466168:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13384.238983] Lustre: 466168:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13384.333383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13384.345156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13384.496629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13384.936981] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13388.519780] LNetError: 463899:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13388.537651] LNetError: 463899:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [13388.550534] LNetError: Protocol error connecting to 192.168.206.155@tcp116 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13388.561971] LNetError: Skipped 12 previous similar messages [13392.661508] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13400.036076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13400.036277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13400.039684] LustreError: Skipped 1 previous similar message [13400.045369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13400.050206] Lustre: Skipped 1 previous similar message [13400.863143] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp124 [13404.580506] Lustre: server umount lustre-OST0000 complete [13410.275214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13410.287751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13410.289362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13410.328992] Lustre: Skipped 2 previous similar messages [13413.962560] Lustre: server umount lustre-MDT0000 complete [13416.849908] LustreError: 463885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407123 with bad export cookie 3411269736394299273 [13416.858121] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13416.859813] LustreError: 463885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13417.138925] Lustre: server umount lustre-MDT0001 complete [13424.352374] LNetError: 463908:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13424.360388] LNetError: 463908:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 25 previous similar messages [13424.369553] LNetError: Protocol error connecting to 192.168.206.155@tcp126 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13424.375829] LNetError: Skipped 25 previous similar messages [13426.429538] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [13428.757524] Key type lgssc unregistered [13429.071105] LNet: 467780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13429.080192] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13429.084305] LNet: Removed LNI 192.168.206.155@tcp [13430.111388] LNet: Removed LNI 192.168.206.155@tcp100 [13430.115320] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13430.123509] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13430.132910] LNet: Removed LNI 192.168.206.155@tcp101 [13431.200255] LNet: Removed LNI 192.168.206.155@tcp102 [13431.201915] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13431.207851] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13431.213151] LNet: Removed LNI 192.168.206.155@tcp103 [13432.223188] LNet: Removed LNI 192.168.206.155@tcp104 [13433.247224] LNet: Removed LNI 192.168.206.155@tcp105 [13433.250249] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13433.256897] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13433.269584] LNet: Removed LNI 192.168.206.155@tcp106 [13434.337796] LNet: Removed LNI 192.168.206.155@tcp107 [13435.361916] LNet: Removed LNI 192.168.206.155@tcp108 [13436.384616] LNet: Removed LNI 192.168.206.155@tcp109 [13437.407292] LNet: Removed LNI 192.168.206.155@tcp110 [13437.417816] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13437.424549] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13437.438741] LNet: Removed LNI 192.168.206.155@tcp111 [13438.496957] LNet: Removed LNI 192.168.206.155@tcp112 [13439.521497] LNet: Removed LNI 192.168.206.155@tcp113 [13440.543179] LNet: Removed LNI 192.168.206.155@tcp114 [13441.567159] LNet: Removed LNI 192.168.206.155@tcp115 [13442.591133] LNet: Removed LNI 192.168.206.155@tcp116 [13443.615139] LNet: Removed LNI 192.168.206.155@tcp117 [13444.639208] LNet: Removed LNI 192.168.206.155@tcp118 [13445.663138] LNet: Removed LNI 192.168.206.155@tcp119 [13445.664749] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13445.684610] LNetError: 467780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13445.701253] LNet: Removed LNI 192.168.206.155@tcp120 [13446.751257] LNet: Removed LNI 192.168.206.155@tcp121 [13447.776937] LNet: Removed LNI 192.168.206.155@tcp122 [13448.800097] LNet: Removed LNI 192.168.206.155@tcp123 [13448.815821] LNet: Removed LNI 192.168.206.155@tcp124 [13449.824796] LNet: Removed LNI 192.168.206.155@tcp125 [13450.847620] LNet: Removed LNI 192.168.206.155@tcp126 [13451.871145] LNet: Removed LNI 192.168.206.155@tcp127 [13452.895210] LNet: Removed LNI 192.168.206.155@tcp128 [13453.919175] LNet: Removed LNI 192.168.206.155@tcp129 [13453.921512] LNet: Removed LNI 192.168.206.155@tcp130 [13453.928146] LNet: Removed LNI 192.168.206.155@tcp131 [13453.935336] LNet: Removed LNI 192.168.206.155@tcp132 [13453.938253] LNet: Removed LNI 192.168.206.155@tcp133 [13453.942564] LNet: Removed LNI 192.168.206.155@tcp134 [13453.960660] LNet: Removed LNI 192.168.206.155@tcp135 [13454.879690] Key type .llcrypt unregistered [13454.882078] Key type ._llcrypt unregistered [13479.508571] Key type ._llcrypt registered [13479.513542] Key type .llcrypt registered [13479.640523] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [13494.812541] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13495.728734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13495.780376] alg: No test for adler32 (adler32-zlib) [13497.052538] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [13497.436133] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13499.128365] Key type lgssc registered [13500.200896] Lustre: Echo OBD driver; http://www.lustre.org/ [13529.436698] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:00:33 (1787407233) [13543.812440] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [13546.414573] Key type lgssc unregistered [13546.676491] LNet: 471869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13546.686515] LNetError: 471869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13546.701270] LNet: Removed LNI 192.168.206.155@tcp [13547.370797] Key type .llcrypt unregistered [13547.375326] Key type ._llcrypt unregistered [13563.182782] Key type ._llcrypt registered [13563.186693] Key type .llcrypt registered [13563.331777] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13564.107095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13564.190177] alg: No test for adler32 (adler32-zlib) [13565.396370] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [13565.698861] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13567.391180] Key type lgssc registered [13569.146761] Lustre: Echo OBD driver; http://www.lustre.org/ [13579.238478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13579.286328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13580.596615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13580.633708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13580.724661] Lustre: lustre-MDT0000: new disk, initializing [13580.845483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13580.865490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13584.468410] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13592.901297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13593.013534] Lustre: 473091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13593.070504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13593.074274] Lustre: Skipped 1 previous similar message [13593.132254] Lustre: lustre-MDT0001: new disk, initializing [13593.229621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13593.275460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13593.282040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13596.489697] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13603.335778] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13607.964422] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13613.103379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13613.302918] Lustre: lustre-OST0000: new disk, initializing [13613.305330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13613.309584] Lustre: 475059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13613.395295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13618.859604] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13621.805378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13621.811646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13621.895542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13626.884230] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13630.536965] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing lnet_if_list [13634.992892] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13635.024351] LNet: Added LNI 192.168.206.155@tcp10 [8/256/0/180] [13640.386804] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13640.421701] LNet: Added LNI 192.168.206.155@tcp11 [8/256/0/180] [13644.508151] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13644.535919] LNet: Added LNI 192.168.206.155@tcp12 [8/256/0/180] [13649.642595] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13649.686032] LNet: Added LNI 192.168.206.155@tcp13 [8/256/0/180] [13654.420635] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13654.453622] LNet: Added LNI 192.168.206.155@tcp14 [8/256/0/180] [13659.146957] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13659.188813] LNet: Added LNI 192.168.206.155@tcp15 [8/256/0/180] [13664.203589] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13664.231344] LNet: Added LNI 192.168.206.155@tcp16 [8/256/0/180] [13668.894970] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13668.924830] LNet: Added LNI 192.168.206.155@tcp17 [8/256/0/180] [13673.734290] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13673.761526] LNet: Added LNI 192.168.206.155@tcp18 [8/256/0/180] [13678.402893] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13678.434120] LNet: Added LNI 192.168.206.155@tcp19 [8/256/0/180] [13682.859632] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13682.887578] LNet: Added LNI 192.168.206.155@tcp20 [8/256/0/180] [13687.831627] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13687.860840] LNet: Added LNI 192.168.206.155@tcp21 [8/256/0/180] [13693.015715] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13693.055934] LNet: Added LNI 192.168.206.155@tcp22 [8/256/0/180] [13698.019875] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13698.055589] LNet: Added LNI 192.168.206.155@tcp23 [8/256/0/180] [13703.484902] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13703.519288] LNet: Added LNI 192.168.206.155@tcp24 [8/256/0/180] [13709.116617] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13709.154693] LNet: Added LNI 192.168.206.155@tcp25 [8/256/0/180] [13714.624356] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13714.657556] LNet: Added LNI 192.168.206.155@tcp26 [8/256/0/180] [13720.428577] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13720.484960] LNet: Added LNI 192.168.206.155@tcp27 [8/256/0/180] [13728.618165] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13728.657208] LNet: Added LNI 192.168.206.155@tcp28 [8/256/0/180] [13735.022736] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13735.069963] LNet: Added LNI 192.168.206.155@tcp29 [8/256/0/180] [13743.039205] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13743.075356] LNet: Added LNI 192.168.206.155@tcp30 [8/256/0/180] [13749.786396] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13749.827616] LNet: Added LNI 192.168.206.155@tcp31 [8/256/0/180] [13756.380820] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13756.430571] LNet: Added LNI 192.168.206.155@tcp32 [8/256/0/180] [13765.317825] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13765.364624] LNet: Added LNI 192.168.206.155@tcp33 [8/256/0/180] [13774.828733] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13774.869990] LNet: Added LNI 192.168.206.155@tcp34 [8/256/0/180] [13782.827923] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13782.854486] LNet: Added LNI 192.168.206.155@tcp35 [8/256/0/180] [13788.359933] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13788.393484] LNet: Added LNI 192.168.206.155@tcp36 [8/256/0/180] [13797.202837] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13797.243326] LNet: Added LNI 192.168.206.155@tcp37 [8/256/0/180] [13805.768756] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13805.796469] LNet: Added LNI 192.168.206.155@tcp38 [8/256/0/180] [13813.631459] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13813.650978] LNet: Added LNI 192.168.206.155@tcp39 [8/256/0/180] [13821.106313] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13821.146361] LNet: Added LNI 192.168.206.155@tcp40 [8/256/0/180] [13828.087833] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13828.114593] LNet: Added LNI 192.168.206.155@tcp41 [8/256/0/180] [13834.978957] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13835.030857] LNet: Added LNI 192.168.206.155@tcp42 [8/256/0/180] [13841.864996] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13841.891201] LNet: Added LNI 192.168.206.155@tcp43 [8/256/0/180] [13847.909824] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13847.937834] LNet: Added LNI 192.168.206.155@tcp44 [8/256/0/180] [13854.837219] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13854.876798] LNet: Added LNI 192.168.206.155@tcp45 [8/256/0/180] [13861.858100] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13861.877486] LNet: Added LNI 192.168.206.155@tcp46 [8/256/0/180] [13868.350581] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13868.384483] LNet: Added LNI 192.168.206.155@tcp47 [8/256/0/180] [13874.582979] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13874.606792] LNet: Added LNI 192.168.206.155@tcp48 [8/256/0/180] [13881.417224] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13881.441364] LNet: Added LNI 192.168.206.155@tcp49 [8/256/0/180] [13889.244694] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13889.273308] LNet: Added LNI 192.168.206.155@tcp50 [8/256/0/180] [13905.892318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13905.911455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13905.929080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13908.451370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13908.453535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13910.685713] Lustre: server umount lustre-MDT0000 complete [13913.785904] LustreError: 473085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407620 with bad export cookie 5325393757746418246 [13913.795122] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13913.802876] LustreError: 473085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13914.175664] Lustre: server umount lustre-MDT0001 complete [13931.186767] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13935.071147] Lustre: 472367:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787407625/real 1787407625] req@00000000b832a349 x1874232367805056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787407641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13935.135036] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13935.144611] Lustre: Skipped 1 previous similar message [13939.167455] Lustre: 472366:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787407630/real 1787407630] req@00000000c33f99dd x1874232367805312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787407646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13939.592542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13944.287278] Lustre: 472365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787407635/real 1787407635] req@00000000e8ed3bff x1874232367805440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787407651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13949.407360] Lustre: 472364:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787407640/real 1787407640] req@00000000429ca5e2 x1874232367805568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787407656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13958.687759] LustreError: 472363:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e8ed3bff x1874232367806208/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13958.915127] LustreError: 475052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13958.928547] LustreError: 475052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13958.961255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13960.992213] LustreError: 499297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13961.005282] LustreError: 499297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13962.986693] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13964.070648] LustreError: 499295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13964.074028] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13964.087574] LustreError: 499295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13969.388711] LustreError: 499295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13969.409303] LustreError: 499295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13970.045447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13970.371651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13973.636637] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13975.532661] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13979.405623] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13983.633343] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14000.097380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14000.113531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14000.140311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14000.149322] Lustre: Skipped 2 previous similar messages [14001.130368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14005.850208] Lustre: server umount lustre-OST0000 complete [14011.360643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14011.361076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14011.361689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14011.361693] Lustre: Skipped 2 previous similar messages [14011.386099] Lustre: Skipped 2 previous similar messages [14015.569426] Lustre: server umount lustre-MDT0000 complete [14016.492851] LustreError: 499301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14016.515343] LustreError: 499301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14018.368424] LustreError: 473083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407725 with bad export cookie 5325393757746420689 [14018.375047] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14018.377611] LustreError: 473083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14018.617436] Lustre: server umount lustre-MDT0001 complete [14027.126563] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14029.368743] Key type lgssc unregistered [14029.776441] LNet: 502075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14029.791439] LNetError: 502075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14029.799762] LNet: Removed LNI 192.168.206.155@tcp [14029.804791] LNet: Removed LNI 192.168.206.155@tcp10 [14029.807804] LNet: Removed LNI 192.168.206.155@tcp11 [14029.811419] LNet: Removed LNI 192.168.206.155@tcp12 [14029.817300] LNet: Removed LNI 192.168.206.155@tcp13 [14029.824357] LNet: Removed LNI 192.168.206.155@tcp14 [14029.828700] LNet: Removed LNI 192.168.206.155@tcp15 [14029.832768] LNet: Removed LNI 192.168.206.155@tcp16 [14029.838723] LNet: Removed LNI 192.168.206.155@tcp17 [14029.844112] LNet: Removed LNI 192.168.206.155@tcp18 [14029.848343] LNet: Removed LNI 192.168.206.155@tcp19 [14029.854289] LNet: Removed LNI 192.168.206.155@tcp20 [14029.858551] LNet: Removed LNI 192.168.206.155@tcp21 [14029.861293] LNet: Removed LNI 192.168.206.155@tcp22 [14029.867125] LNet: Removed LNI 192.168.206.155@tcp23 [14029.872088] LNet: Removed LNI 192.168.206.155@tcp24 [14029.876101] LNet: Removed LNI 192.168.206.155@tcp25 [14029.880493] LNet: Removed LNI 192.168.206.155@tcp26 [14029.886379] LNet: Removed LNI 192.168.206.155@tcp27 [14029.890466] LNet: Removed LNI 192.168.206.155@tcp28 [14029.895272] LNet: Removed LNI 192.168.206.155@tcp29 [14029.900570] LNet: Removed LNI 192.168.206.155@tcp30 [14029.905206] LNet: Removed LNI 192.168.206.155@tcp31 [14029.910047] LNet: Removed LNI 192.168.206.155@tcp32 [14029.916092] LNet: Removed LNI 192.168.206.155@tcp33 [14029.921209] LNet: Removed LNI 192.168.206.155@tcp34 [14029.926375] LNet: Removed LNI 192.168.206.155@tcp35 [14029.931417] LNet: Removed LNI 192.168.206.155@tcp36 [14029.937480] LNet: Removed LNI 192.168.206.155@tcp37 [14029.942330] LNet: Removed LNI 192.168.206.155@tcp38 [14029.947086] LNet: Removed LNI 192.168.206.155@tcp39 [14029.951356] LNet: Removed LNI 192.168.206.155@tcp40 [14029.954380] LNet: Removed LNI 192.168.206.155@tcp41 [14029.957575] LNet: Removed LNI 192.168.206.155@tcp42 [14029.961217] LNet: Removed LNI 192.168.206.155@tcp43 [14029.965327] LNet: Removed LNI 192.168.206.155@tcp44 [14029.969292] LNet: Removed LNI 192.168.206.155@tcp45 [14029.973635] LNet: Removed LNI 192.168.206.155@tcp46 [14029.977980] LNet: Removed LNI 192.168.206.155@tcp47 [14029.981662] LNet: Removed LNI 192.168.206.155@tcp48 [14029.985586] LNet: Removed LNI 192.168.206.155@tcp49 [14030.001996] LNet: Removed LNI 192.168.206.155@tcp50 [14030.671884] Key type .llcrypt unregistered [14030.674256] Key type ._llcrypt unregistered [14049.859913] Key type ._llcrypt registered [14049.861944] Key type .llcrypt registered [14049.960927] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [14064.696869] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14065.741204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14065.784179] alg: No test for adler32 (adler32-zlib) [14066.810937] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [14067.114926] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [14068.760160] Key type lgssc registered [14069.925265] Lustre: Echo OBD driver; http://www.lustre.org/ [14101.175218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14101.195839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14102.389779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14102.442119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14102.525625] Lustre: lustre-MDT0000: new disk, initializing [14102.618804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14102.645275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14106.696175] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14117.080818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14117.201944] Lustre: 505270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14117.271896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14117.275712] Lustre: Skipped 1 previous similar message [14117.335439] Lustre: lustre-MDT0001: new disk, initializing [14117.425988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14117.453193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14117.471250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14121.450698] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14130.495808] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14136.567265] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14144.640196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14144.960322] Lustre: *** cfs_fail_loc=90f, val=0*** [14145.096284] Lustre: lustre-OST0000: new disk, initializing [14145.099821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14145.108754] Lustre: 507286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14145.241628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14152.037441] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14154.323888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14154.338967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14154.459164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14160.308589] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14170.096414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14170.097174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14170.131670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14170.137653] Lustre: Skipped 1 previous similar message [14173.869372] Lustre: server umount lustre-OST0000 complete [14178.785982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14178.792135] LustreError: Skipped 1 previous similar message [14178.794493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14178.804032] Lustre: Skipped 1 previous similar message [14178.808623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14180.320607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14180.323042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14183.709338] Lustre: server umount lustre-MDT0000 complete [14186.916577] LustreError: 505262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407893 with bad export cookie 11525818492122010779 [14186.923242] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14186.927520] LustreError: 505262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14187.306159] Lustre: server umount lustre-MDT0001 complete [14195.476593] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14197.591638] Key type lgssc unregistered [14197.863348] LNet: 509192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14197.872092] LNetError: 509192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14197.885038] LNet: Removed LNI 192.168.206.155@tcp [14198.571059] Key type .llcrypt unregistered [14198.572626] Key type ._llcrypt unregistered [14214.562669] Key type ._llcrypt registered [14214.565725] Key type .llcrypt registered [14214.698527] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14215.689191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14215.759921] alg: No test for adler32 (adler32-zlib) [14216.881567] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [14217.091967] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [14218.807453] Key type lgssc registered [14220.291960] Lustre: Echo OBD driver; http://www.lustre.org/ [14228.101301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14228.121167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14229.688812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14233.775283] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14239.214384] LustreError: 510442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14239.245443] LustreError: 510442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14241.902205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14242.282677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14245.643280] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14252.413883] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14257.548760] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14263.918887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14264.146964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14270.653098] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14276.980463] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14286.736117] loop0: detected capacity change from 0 to 204800000 [14286.865282] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14286.875535] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14286.890873] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14291.403216] loop0: detected capacity change from 0 to 204800000 [14291.596327] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14291.606657] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14291.618763] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14295.757544] loop0: detected capacity change from 0 to 204800000 [14300.312020] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14300.437713] Lustre: fs15246-OST0000: new disk, initializing [14300.441395] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14300.447601] Lustre: 514033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14300.497674] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14305.994430] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14314.059454] loop1: detected capacity change from 0 to 204800000 [14318.636462] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14318.737088] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14318.799641] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14318.917933] Lustre: fs15246-MDT0000: new disk, initializing [14319.062548] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14319.090669] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14319.179112] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14320.226826] Lustre: 515155:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004d9bf35c x1874233050311168/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787408043 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14322.873091] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14325.284660] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14325.291238] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14325.345546] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14344.693208] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14344.717809] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14349.752881] Lustre: server umount fs15246-MDT0000 complete [14349.814166] LustreError: 510438:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14354.918446] LustreError: 510438:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14358.727208] Lustre: server umount fs15246-OST0000 complete [14369.761994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14369.772339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14369.793473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14375.225893] Lustre: server umount lustre-OST0000 complete [14380.516215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14380.516539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14380.519042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14380.519050] Lustre: Skipped 2 previous similar messages [14380.531626] Lustre: Skipped 2 previous similar messages [14384.798972] Lustre: server umount lustre-MDT0000 complete [14385.632505] LustreError: 510443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14387.971144] LustreError: 510422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408094 with bad export cookie 7778684515941309294 [14387.975243] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14387.981135] LustreError: 510422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14388.289165] Lustre: server umount lustre-MDT0001 complete [14397.949787] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14400.534807] Key type lgssc unregistered [14400.796933] LNet: 518251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14400.802561] LNetError: 518251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14400.822264] LNet: Removed LNI 192.168.206.155@tcp [14401.576944] Key type .llcrypt unregistered [14401.579361] Key type ._llcrypt unregistered [14417.907232] Key type ._llcrypt registered [14417.909129] Key type .llcrypt registered [14418.015835] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14418.902740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14418.998733] alg: No test for adler32 (adler32-zlib) [14420.030335] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [14420.264433] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [14421.943287] Key type lgssc registered [14422.962693] Lustre: Echo OBD driver; http://www.lustre.org/ [14456.176540] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [14465.056246] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14497.677489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14497.724595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14498.997369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14499.026536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14499.099984] Lustre: lustre-MDT0000: new disk, initializing [14499.179619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14499.197982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14503.331525] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14513.095593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14513.261813] Lustre: 522716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14513.351828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14513.355265] Lustre: Skipped 1 previous similar message [14513.425198] Lustre: lustre-MDT0001: new disk, initializing [14513.528833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14513.562680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14513.572614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14518.085175] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14527.544407] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14532.609925] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14538.514318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14538.801655] Lustre: lustre-OST0000: new disk, initializing [14538.807968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14538.814628] Lustre: 524683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14538.904769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14543.661487] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14548.004445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14548.012597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14548.140159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14552.099026] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14556.054677] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14556.305210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14560.756893] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14560.980433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14563.299362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14563.309059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14563.325776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14563.814218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14563.815893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14563.839027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14568.938636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14574.054816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14574.070269] Lustre: Skipped 2 previous similar messages [14577.119372] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14577.292468] Lustre: server umount lustre-OST0000 complete [14584.288900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14584.302210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14584.311365] Lustre: Skipped 1 previous similar message [14584.329375] Lustre: Skipped 1 previous similar message [14586.914520] Lustre: server umount lustre-MDT0000 complete [14590.025905] LustreError: 522708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408296 with bad export cookie 10069791303502529302 [14590.033771] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14590.037438] LustreError: 522708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14590.595422] Lustre: server umount lustre-MDT0001 complete [14614.984875] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [14623.422947] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14652.444499] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14369 sec ========= 10:19:16 (1787408356) [14654.072060] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:19:18 (1787408358) === [14657.315295] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:19:21 (1787408361) === [14689.771819] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14691.963581] Key type lgssc unregistered [14692.222545] LNet: 531327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14692.230329] LNetError: 531327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14692.245504] LNet: Removed LNI 192.168.206.155@tcp [14693.005914] Key type .llcrypt unregistered [14693.007818] Key type ._llcrypt unregistered